900 years ago, the founder of House Frey was given lands and titles. He judged that he could have wealth and power if he built a bridge to cross the Green Fork, a river that cut off the North from the South. It took the Frey's three generations of building to complete the founder's vision. The bridge was large, smooth stone, made possible for two wagons to pass abreast. Then, they put formidable keeps on either side, with stone walls, portcullises, and heavily fortifeid gates along with murder holes and arrow slits to ensure that no one crossed without paying the toll. A stone building was put in the middle of the bridge called the Water Tower.
The plan worked, and the Frey's of the Crossing grew rich, exacting tolls upon many lords who despised the family for the way they got powerful through cunning, calling them upstarts.
After the Red Wedding, where Walder Frey and the Bolton's of the Dreadfort murdered the King in the North, Robb Stark, in the War of the Five Kings, the Frey's fell apart. Walder Frey died of old age and almost all of the Frey's backstabbed each other until there were only few remaining, others were killed off by the northerners. The few living surrendered to Aegon VI and continued their line as Lords of the Crossing.
Under the rule of Edwyn Frey, the current lord of the crossing, the Frey's have gotten even richer. His toll was the highest out of any of the Frey's before him, to make up for his father's failure as lord. The Frey's are often distrusted and despised still, some by northerners who remember the Red Wedding, and some by men who believe that Edwyn Frey is trying to backstab his way to Lord Protector of the Riverlands.