The answer may vary by school and is likely governed by the school's IFC (Inter-Fraternity Council) or Panhellenic Council (for sororities). When you pledge a fraternity/sorority, you do sign a formal document that is set forth by IFC/Panhel. That document probably establishes the financial guidelines, with the amount set by the individual fraternity.
That said, most frats/schools assess and pay dues by semester. So if you formally resigned the fraternity, I would assume that means you're out the dues for the current semester but nothing going forward. Not sure how the money would be handled if you quit during pledging but I would guess it's a similar scenario.