Gensler subsequently apologized to Lew for the outburst. He also softened his demands, cutting a deal with the European Union (EU) a week later. Gensler, chairman of a historically obscure agency called the Commodity Futures Trading Commission (CFTC), had again pushed an idea to the brink until forced to settle.

The fate of one of Gensler's central goals shows why the U.S. attempt to rein in the world's most secretive and profitable financial products falls short of the vision he promoted four years ago. While he won regulators the power to reach deep into a $633 trillion market, Wall Street preserved its dominance in derivatives trading with one of the largest sustained lobbying attacks on a single Washington agency.

In the end, the full force of the rules that the CFTC is writing under the authority of the 2010 Dodd-Frank financial regulatory law will apply to only a small share of the global market—possibly less than 20 percent, according to data compiled by Bloomberg.

Whole segments of the business have been carved out of the rules. Derivatives based on foreign-exchange rates are largely exempt. Some firms are modifying their products to escape new oversight. And after Gensler's compromise with Europe, American banks will be able to sell derivatives overseas without direct U.S. scrutiny.

Derivatives were famously labeled “financial weapons of mass destruction” by investor Warren Buffett before they became accelerants in the 2008 meltdown and led to the $182 billion U.S bailout of AIG. The insurer couldn't cover credit-default swaps it sold to banks when the global squeeze began. The derivative rules now taking shape are a core element of Dodd-Frank, the government's biggest foray into regulating the financial industry since the SEC was created during the Great Depression.

The stakes are high, both for big banks, which have earned more than $50 billion a year dealing in derivatives, and for the larger economy. While they are rarely traded by individuals, many businesses and institutions use derivatives in investment strategies, which affect heating-oil prices, college savings funds, and highway projects. Dairy farmers buy them to protect against sudden price increases in corn and soybean meal for their cows, helping to keep milk prices stable for consumers.

The CFTC has produced about 60 rules for derivatives. This story reviews how the banking industry and its allies forced Gensler to retreat on the three most consequential ones for Wall Street. It is based on agency, court, and congressional records; private notes, e-mails, and documents; and interviews with dozens of executives, lobbyists, and U.S. officials, some of whom spoke on condition of anonymity to describe private meetings.

Lobbying Blitz

When the standing of finance lobbyists in Washington began to rebound along with the economy their industry sent into recession, they sought to influence the language in the CFTC's intricate rulebook. They flooded the agency with visits, unleashed thousands of comment letters, enlisted sympathetic lawmakers, and stoked differences between the CFTC and SEC.

Under the three-year assault, the CFTC created winners and losers with keystrokes. Changing a single number in one rule undermined potential competition to banks. Another tweak allows firms including Koch Industries Inc. and ConocoPhillips to trade billions of dollars in swaps and avoid the most stringent rules.

“The banks are going to be fine,” said Sunil Hirani, chief executive officer of trueEX Group LLC, who helped pioneer electronic trading of derivatives. “They are going to make a ton of money.”

Jill E. Sommers, a Republican CFTC commissioner who stepped down in July, said the outcome leaves the big players in charge.

“Looking back, of course with 20-20 hindsight, I wish we would have done more to encourage competition,” she said. “The only people that can afford to stay in the business are the people who have already long established their footprint in the market.”

Pride of authorship helps explain why Gensler fought so hard. It's been a closely held secret that he was the invisible hand behind Dodd-Frank's derivatives section. His biggest coup was to slip carefully crafted language into the law that the banks didn't initially realize could give the CFTC authority over their foreign branches.

Gensler turned his agency upside down trying to preserve the intent of his own text. His demanding style got results, while also driving aides to a point of exhaustion and sending some running for the exits. He alienated other commissioners and angered what he calls the G-16—the 16 big banks that, according to a 2010 Citigroup Inc. report cited by Deloitte LLP, earn about $55 billion a year from derivatives.

In the end, Gensler also sacrificed his ambition, creating so many political enemies in Congress and the industry that his chances for a second term as chairman are dim.

“He burned a lot of bridges getting it all done, within the commission and internationally,” said David Hirschmann, president of the Center for Capital Markets Competitiveness at the U.S. Chamber of Commerce.

Gensler, 55, is hustling to finish the rules this year before his tenure at the CFTC ends Dec. 31. He says any shortcomings pale next to Dodd-Frank's core accomplishment: Many derivative trades once handled privately are being forced into the open. The market will be cheaper for buyers and safer for the economy because participants will report transactions and clear them through a third party, putting up collateral.

Even that success may have unintended consequences. Some finance scholars, Wall Street banks, and Gensler himself have warned that concentrating trades at a few big clearinghouses that settle trillions of dollars in deals creates a new risk—a potential too-big-to-fail powder keg when the next crisis hits.

Thousand Visits

The lobbying blitz pitted Gensler against banks, including the one where he started his career as a young star—Goldman Sachs Group Inc., which made more than 150 visits and calls to the agency from April 2010 through July 2013, an analysis of CFTC records shows.

Representatives of Gensler's G-16 opponents together made more than 1,000 contacts with the agency, the records show. They included lobbyists, lawyers, and bank presidents. Alone or in groups, they attended more than a third of the 2,100 sessions that the CFTC's staff reported holding with outsiders.

All told, more than 3,500 people joined meetings at the agency's red-brick headquarters just outside Washington's K Street lobbying corridor.

The lineup shows how derivatives have penetrated economic life beyond Wall Street. Royal Dutch Shell Plc and Caterpillar Inc. sent emissaries. The American Bankers Association came calling, as did the American Bakers Association, Chicago Mayor Rahm Emanuel, the head of Carolina Cotton Growers Cooperative Inc., the director of fuel for AirTran Holdings Inc., and the treasurer of Walt Disney Co. (Among firms lobbying was Bloomberg LP, parent of Bloomberg News, which started an electronic derivatives-trading platform that competes with banks. The firm also filed an unsuccessful federal lawsuit against the CFTC involving its trading rules.)

The banking industry was the most persistent and well-financed of the visitors. In the first year after Dodd-Frank was enacted, Wall Street's biggest lobbying group—the Securities Industry and Financial Markets Association, known as Sifma—paid more than $3 million to law firms working on regulations, public filings show. Individual banks spent millions more.

Sifma President Kenneth Bentsen said the industry's dealings with Gensler aren't about “a win or lose.” Banks that bear the burden of new rules have a responsibility to make sure lawmakers and regulators understand their perspective, he said.

“None of these things are as simple or as crystal clear as they were sold to be,” said Bentsen. “Gary is a very smart individual. He obviously was in the industry. He has been in government. He is very certain of his view of things and is very determined. That's what guides him. But there is more than one view.”

Seminal Meeting

Gensler's vision took shape on a January day in 2009. He sat in a glass-walled conference room across from two other people President-elect Barack Obama picked to rebuild the discredited U.S. oversight system for Wall Street: Timothy F. Geithner, who would be taking over the Treasury, and Mary Schapiro, nominated for chairman of the SEC.

Just as the 2001 terrorist attacks spawned a vast new national-security complex, the credit crisis was fueling calls for a regulatory wall against future financial disasters. All three agreed the U.S. had to wrest control from firms including Goldman Sachs, the biggest securities firm that was transformed into a bank, and JPMorgan Chase & Co., the biggest U.S. bank by assets, both of which made billions selling derivatives behind closed doors before taking taxpayer bailouts in 2008.

Gensler wanted trades to be public, buyers forced to post collateral, and new businesses set up to compete with banks.

“We weren't staffed,” Gensler said, describing the meeting in an interview. “There were no briefing books.” It was a “vision that kind of we cobbled together.”

Unlike regulators who have to learn the language of Wall Street, Gensler was a native speaker. With an MBA from the University of Pennsylvania's Wharton School, he became one of the youngest partners in Goldman Sachs history. He quit at 39, with investments he recently reported to be worth more than $15 million, and joined Treasury in 1997 as an assistant secretary.

Gensler is more self-assured than the typical finance nerd. He's fond of kicking off his shoes in the office. Unwinding after an industry conference in Florida last March, he pulled a female attendee onto the dance floor and gyrated expertly to pop rock tracks. A video clip went viral among Washington insiders, including surprised CFTC staff members.

Gensler also surprised Wall Street with his two-step on derivatives. At Treasury, he advocated for their deregulation. He said he assumed that since banks were heavily regulated as institutions it would be redundant to make rules for each of their activities. “That was a bad assumption,” he said.

Derivatives and the debate over their effects on society are at least as old as capitalism. Some scholars trace them to about 580 B.C., noting that Greek philosopher Thales the Milesian was said to have gotten rich buying rights to purchase olive presses after discerning a big harvest was on the way.

A swap, a common form of derivative, is at its simplest a contract between two parties who agree to exchange money or goods depending on what happens to an asset. Swaps are often used to hedge risk. An investor in a company's bonds might buy a credit-default swap, which provides a kind of insurance against the chance the firm can't repay its debt.

AIG Bailout

By the late 1990s, the U.S. swaps market had grown beyond basic hedging into a casino of increasingly ingenious and interconnected products.

When Gensler met with Geithner and Schapiro at the Obama transition offices, set up in a glass-and-concrete office building between the U.S. Capitol and the White House, the government was well into the AIG bailout. Gensler wanted the new regulatory machine to reach beyond the credit-default swaps sold by AIG into all derivatives that aren't traded on exchanges, the kind known as over-the-counter, or OTC, swaps.

“This would be the entire product suite,” he recalled saying.

To Geithner, Gensler's views made sense. The Treasury chief had examined the derivatives market in 2005 and 2006 while he was head of the Federal Reserve Bank of New York. He saw risk-management practices that were stuck in the dark ages—trades confirmed with paper, pencils, and faxes—and had virtually no oversight, according to a person familiar with his thinking.

Within months of the downtown Washington meeting, the Obama administration issued a proposal for “comprehensive regulation” of derivatives. News of that plan roiled the banking industry, which had long argued that swaps increase the flow of credit and reduce risk by spreading it around.

With few exceptions, Washington had sided with the banks over the years. “Regulation that serves no useful purpose hinders the efficiency of markets to enlarge standards of living,” then-Federal Reserve Chairman Alan Greenspan told Congress in 1998, explaining why swap rules weren't necessary.

When regulation talk resurfaced, banks went looking for help again.

There is a “reasonably high risk that the OTC derivatives market could be destroyed,” Blythe Masters, then chairman of Sifma, said at a private April 2, 2009, meeting where Bank of England Deputy Governor Paul Tucker was also in attendance.

Masters had helped JPMorgan create the credit derivatives market in the 1990s. She asked Tucker how banks could counter the “increasingly prevailing view within Washington DC and Main Street” that swaps threaten the financial system, according to minutes given to Sifma's board members. Tucker suggested the industry “challenge the notion that government knows best and should engage the academic community,” the minutes show.

Masters now oversees JPMorgan's commodities unit, which agreed to pay $410 million in July to settle claims it manipulated wholesale electricity markets. She declined through a spokesman to comment. Tucker didn't respond to requests for comment sent to the Bank of England.

Rebuffing Bankers

Bankers who told each other Gensler's resume would make him sympathetic were soon disappointed. It had been more than a decade since he'd departed Goldman Sachs. He left the Treasury in 2001 when Republican George W. Bush became president. In the run-up to the global financial crisis he was largely focused on being a stay-at-home father for his three daughters and a caregiver for his wife, an artist. She died of cancer in 2006.

In the intervening years, his views had changed. In January 2010, Gensler accepted an invitation to lunch at New York's Waldorf Astoria hotel with executives from Goldman Sachs, Credit Suisse Group AG, Deutsche Bank AG, Bank of New York Mellon Corp., and others. He rebuffed their concerns, saying his duty now was to taxpayers, according to people who attended. Asked to name the main obstacle to an improved system, the people said, he gestured at his hosts and replied: “You.”

Congress was already at work on the Obama administration's oversight plan for Wall Street, which reimagined regulation in areas including capital requirements, consumer finance, and mortgages. The derivatives piece landed in the Senate Agriculture Committee. During drafting sessions, Gensler sometimes sat at the table reserved for staff, advising its Democratic chairwoman, Blanche Lincoln of Arkansas.

Gensler was reflecting a changed climate in Washington, where Wall Street wasn't welcome on the front lines in Congress as investigators probed allegations of wrongdoing during the financial crisis. The industry that wanted to fight “tooth and nail” against swap rules turned to others to make its case, said Michael Barr, then an assistant Treasury secretary and an Obama administration point man on Dodd-Frank.

“They brought in CEOs and treasurers of Fortune 500 companies to say how great it was for America that there was this closed dealer market,” Barr said.

Industrial firms such as Deere & Co. told lawmakers their balance sheets would crater if they had to post collateral on trades hedging costs of things like fuel. They secured one of the few exemptions written in the law. Since these so-called end users usually buy swaps from banks, the carve-out protects one Wall Street profit center.

The core financial industry won little in the drafts of Dodd-Frank, thanks in part to Gensler. As lawmakers debated the bill, he took sections home Friday nights. His aides would wake up on Saturdays to e-mails with suggestions and questions that he often expected them to deal with before Monday morning, according to former and current employees.

Dan Berkovitz, the CFTC general counsel at the time, and John Riley, the agency's head of legislative affairs, sent text and suggestions to Democratic staff members on the House Financial Services and Agriculture committees that ended up in the bill almost verbatim, e-mails show.

“Here's the cheat sheet,” Riley wrote as he offered advice to a House staff member in October 2009. In January 2010, he gave Senate aides Gensler's thinking on trading rules, adding that a “more formal legislative proposal” would follow.

Gensler campaigned publicly and behind the scenes. He published a Feb. 24, 2010, column in the Financial Times titled “How to stop another derivatives inferno.” He forwarded the piece to Obama, with a note telling the president that two top priorities—plugging the end-user loophole and making trades more transparent—“are being weakened as a result of opposition from Wall Street and corporate interests,” according to a copy of the memo attached to an e-mail to Congress.

In June 2010, as House and Senate lawmakers negotiated the final language in Dodd-Frank, Gensler was seen by reporters padding around in his stocking feet for the late-night sessions. He whispered in Lincoln's ear to ward off last-minute changes.

Hours after Obama signed Dodd-Frank that July, Gensler invited Capitol Hill staff members who had worked on the derivatives section to a briefing. The 18 months he had spent pressing his case had paid off: Most of what he and the administration had wanted, down to exact language in many cases, was now enshrined in law. “I felt terrific,” Gensler said.

Annette Nazareth, one of the finance industry's leading Washington lawyers, heard about Gensler's briefing. The meeting was to be “followed by a cocktail party as a 'thank you.' How gracious,” she wrote in mock admiration in an e-mail to a friend who worked at the SEC on July 22, 2010.

Gensler is “the James Brown of the regulatory agencies,” Nazareth added, comparing him to the legendary hardest-working man in show business.

The celebration was short. Gensler had to defend his congressional victories.

An agency that might only write five rules in a year was now going to do 60. He kicked the CFTC staff into high gear, reorganizing them into teams around each rule. He exiled team leaders who didn't share his mission or perform to his standards, people with knowledge of the process said.

He ran the show more like a Goldman Sachs operator than a government bureaucrat. After he pledged to hold a transparent and cooperative process, other commissioners felt cut out of the communication loop. They were frustrated and angry when Gensler showed them complex proposals late in the process when he had been discussing them for weeks with the staff, according to former and current CFTC employees.

Staff members unaccustomed to working long hours, nights, and weekends would occasionally sleep on the couches on the ninth floor, where the commissioners had their offices, the people said. With the industry arming for a major pushback, Gensler told them, speed and dedication were essential.

An early battle revolved around a number—one that the CFTC wanted to write into the rules to inject competition into the swaps business.

Slicing Profits

Gensler's idea was to crack open a closed system and create an electronic market more like futures or stocks. Under existing procedures, someone wanting to buy a swap was free to contact a single broker, often a large bank, and make a deal over the phone that would stay secret.

Big banks and brokers favored the status quo. If forced to show their cards to the entire market, others could see their bets and undermine their strategies. Their cut of the deals—and profits—might also come under scrutiny from customers and regulators. Electronic trading and other rules could slice bank revenue from OTC swaps by 35 percent, from $33 billion to $21.2 billion, the Deloitte study said.

Gensler wanted many of the electronic transactions to take place on a new kind of platform, competing with banks and with futures exchanges owned by CME Group Inc. of Chicago and Atlanta-based IntercontinentalExchange Inc. Publicly visible prices on these Swap Execution Facilities, or Sefs, would shift the advantage “from Wall Street to Main Street,” Gensler said.

How many bids would a swap buyer have to solicit? That was the number that set off a two-year fight between Gensler and the industry.

The topic was discussed more frequently than any other issue on the CFTC's agenda. It came up during at least 375 meetings with outsiders, according to data assembled from published CFTC records.

One of the lobbyists most often present at those meetings was Micah Green of Patton Boggs LLP, records show. His firm was hired by the Wholesale Markets Brokers' Association Americas, a trade group that includes swap brokers GFI Group Inc. and ICAP Plc, which reported spending more than $1 million on legal and advocacy work in the year ending June 2012. Green declined to comment. Along with Green, ICAP's representatives were the top four visitors on the issue. Their main concern was preserving trading by phone.

Lobbyists also dragged out consideration of the rule by encouraging complaints from Capitol Hill, exploiting differences among the CFTC's commissioners, and seeking help at the SEC. Some firms that mulled opening Sefs drifted away during the delay.

At first, Gensler championed a fully public and electronic system, in which traders would only rarely be able to cut deals in private on the phone, according to a summary he released in late 2010.

Scott O'Malia, a Republican commissioner, sided with industry arguments that Gensler's plan went beyond the CFTC's authority under Dodd-Frank and wasn't flexible enough for swaps that don't trade as often as futures. The market needs time for “a transition or an evolution,” he said in an interview.

Gensler backtracked. He agreed that not everyone in the market needed to be asked for a bid. Instead, a swap buyer would have to request a set number of bids.

Gensler proposed that the number be set at five.

Immediately, banks and big swap buyers such as money management firms BlackRock Inc. and Vanguard Group Inc. set out to cut the number. One tactic was to encourage a split with the SEC, which regulates about 5 percent of the swaps market and had to issue its own Dodd-Frank trading rules.

The SEC was receptive. The agency had “a higher level of comfort than the CFTC” with letting market participants decide how many bids would be ideal, according to then-SEC chairman Schapiro. Her agency already oversaw off-exchange trading in equity and bond markets.

Gensler's number felt arbitrary, Schapiro said in an interview. “How do you pick five?” Schapiro said. “Why not 10? Why not 3?”

The SEC torpedoed the CFTC's idea, proposing that the number be reduced to just one.

Meanwhile, the CFTC stoked interest in the new system. In March 2011, it staged an event dubbed Sef-a-Palooza. Representatives from two dozen firms traveled to Washington to show their wares to a standing-room-only crowd. Among them were Tradeweb Markets LLC, MarketAxess Holdings Inc., and GFI Group Inc. (Also presenting was Bloomberg LP, parent of Bloomberg News, which has established a Sef.)

The CFTC predicted as many as 40 Sefs would be set up as competitors to the banks and exchanges.

Hirani, the derivatives pioneer who had a walk-on role in the 2010 film “Wall Street: Money Never Sleeps” and was one of the first to trade credit swaps electronically, initially was caught up in the hoopla and planned to start a Sef.

In time, he changed his mind. “I thought it would take too long based on our judgment of the lobbying that market participants would unleash,” Hirani said.

His prediction was borne out. Using the SEC's proposal as a cudgel, banks including Goldman Sachs, JPMorgan, and Deutsche Bank kept showing up at the CFTC arguing to drop the minimum entirely, records show.

On Capitol Hill, where banks were regaining influence as the credit crisis waned, the lobby found sympathizers in both parties, including Representative Barney Frank, the Massachusetts Democrat who was one of the bill's namesakes. At a November 2011 hearing, he called the CFTC's proposal “more intrusive and more complex than was necessary.”

The swing vote on the five-member CFTC was Mark Wetjen, a Democrat and former policy adviser to Senate Majority Leader Harry Reid. Wetjen, who joined the agency in October 2011, had little experience with swap rules while working on Capitol Hill. In February 2013, he floated a plan for two bids. “It was the asset managers and hedgers who were most convincing, and they wanted it flexible,” he said in an interview.

The debate raged in mid-March when bankers and regulators gathered for an annual industry conference in Florida at the pink-stucco Boca Raton Resort and Club—the same meeting where Gensler took his turn on the dance floor.

By the last week of April, Gensler realized the fight was over. He accepted Wetjen's plan for two bids. His only consolation: The limit would rise to three bids after a year.

“I would have preferred to keep it closer to the five,” Gensler said in the interview.

Bart Chilton, a Democratic member of the CFTC, said he voted reluctantly for the compromise. “We certainly did only the bare minimum of what the drafters of Dodd-Frank envisioned,” Chilton said in an interview. “In the end, I think we caved in order to finalize the rule.”

Avoiding Labels

There was another number regulators had to pick: What amount of derivatives could a firm sell before it was big enough to be formally designated a swap dealer? That would mean the government would be constantly looking over its shoulder.

The task had its origins in a financial crisis a decade earlier. A small power company born in Nebraska transformed itself into a massive derivatives trading firm that was allowed to operate with almost no CFTC oversight. It became Enron Corp.

While Enron's 2001 collapse was fueled more by accounting fraud than derivatives, government officials had been seeking for years to ensure that no company could escape regulation by essentially pretending to be in a less-risky business.

Gensler's initial plan would slap the label “swap dealer” on anyone selling at least $100 million worth a year. That would capture dozens of firms far from Wall Street.

Energy executives who said they expected to escape Dodd-Frank by winning the end-user exemption from Congress rushed to get the $100 million threshold raised. Their yearlong campaign involved newly formed industry groups that wouldn't disclose members, well-timed campaign contributions and another back-door assault by the SEC.

To see that the business went far beyond banks, all regulators had to do was to scan the roster of the leading trade organization for derivatives. Among the biggest members of the International Swaps and Derivatives Association were BP Plc, Shell, and Koch Industries, which use swaps to hedge the price of oil and natural gas.

Over the past three decades, these energy companies and others built up side businesses selling derivatives to customers. Koch, for example, has said it was the first dealer of a swap to a major airline looking to hedge fuel costs.

The energy companies said they weren't banks and shouldn't be treated as if they were. Gensler saw it differently: The more of the business that remained outside the stiffest regulations, the more chance of another blow-up. He began calling it the “BP loophole.”

When meeting with lobbyists and industry lawyers, Gensler was quick to show off his mastery of market details, which left some visitors with the sense he had already formed his plan and wasn't interested in their suggestions, according to multiple participants.

Terrence Duffy, executive chairman of CME Group, found Gensler to be different overall than previous CFTC chairmen. “They had a more collegial approach towards making sure the market had an understanding of what they were thinking,” he said in an interview.

Red Font

Early on in the swap-dealer debate, David Perlman of Bracewell & Giuliani LLP in Washington, a former chief counsel of Lehman's commodity business, emerged as a leader of the opposition to Gensler. His Coalition of Physical Energy Companies included Shell, Apache Corp., and NRG Energy Inc.

Perlman rewrote the text of Gensler's proposal and sent it to the CFTC in early 2011. His version raised the $100 million bar to $3 billion. He crossed out the agency's number in red font and put his in blue so no one could miss it, according to a copy later posted on the agency's website.

The $3 billion threshold could help get many of his clients out from under the rules. “The consequence of crossing that line is significant,” he said in an interview.

Another trade group, the Commodity Markets Council, suggested to the CFTC that the cutoff be jacked up to $10 billion.

BP, Koch, Shell, Vitol Group, ConocoPhillips, and Constellation Energy Group Inc. joined together in yet another association—a working group of commercial energy firms, according to public records and people with knowledge of the group's members, which haven't been publicly disclosed.

The group hired Sharon Brown-Hruska, a former acting chairman of the CFTC now with NERA Economic Consulting. She co-wrote a paper concluding that the agency's plan to oversee energy firms as swap dealers would be “very harmful” for consumers. Republicans on Capitol Hill e-mailed it to reporters.

Brown-Hruska said in an interview that the study was effective because it “focused on empirical analysis.” The CFTC proposal “just layered on costs, and it would discourage legitimate and important hedging activity by those firms,” she said.

Gensler wouldn't budge from his $100 million.

After Republicans became a majority in the House in 2011, their staff members met privately with financial lobbyists in the Rayburn office building. They strategized on ways to persuade Democratic members to sign onto bills aimed at slowing Gensler down, two people with knowledge of the sessions said.

In October, Republican legislators wrote a bill that would order a $3 billion threshold for swap dealers. A few Democrats were on board, including Dan Boren of Oklahoma and Mike Ross of Arkansas. While never considered by the full House, it gave lawmakers a chance to hold a public hearing, during which they slammed Gensler's plan.

One company that testified in support of the legislation was Constellation Energy. The political action committees of the Baltimore utility and the company that was in the process of acquiring it—Chicago-based Exelon Corp.—together became among the largest contributors to the 2012 campaign of Representative Randy Hultgren, an Illinois Republican who was the bill's sponsor. They gave a total of $12,500, federal election records show.

Hultgren said in a statement that he backed the bill to protect a “diverse group of Main Street end-users” that he said included Exelon and farmers' cooperatives from “overly broad and burdensome rules that would threaten consumers with higher prices.”

Paul Elsberg, a spokesman for Exelon, said the firm backs candidates “who we believe will support sound energy policies” for Illinois and the nation. “Dodd-Frank should not impose significant costs or restrictions on the ability of companies like ours to manage risk and to hedge our physical exposure to physical commodity prices,” Elsberg said.

At the SEC, industry arguments found a receptive audience in Daniel Gallagher, the newest commissioner and an avowed free-market Republican. SEC staff lawyers already were bucking Gensler, proposing a $1 billion threshold. Gallagher wanted it higher, at $10 billion, to capture only big Wall Street players.

Working with the SEC staff, Gallagher produced an alternative plan. It eventually called for an $8 billion level and a compromise that would drop it to $3 billion after a few years.

Gallagher worked quietly to sell the idea to his counterparts at the CFTC, according to people at both agencies. He invited Republican commissioners Sommers and O'Malia to lunch and spoke with Wetjen by phone, the people said.

'Little Caper'

Gensler got wind of Gallagher's campaign and called him in April 2012, a week before the agencies were scheduled to vote on the rule, said a person with knowledge of the conversation. The two men had been on friendly terms when both commuted by Amtrak between Washington and Baltimore. They hadn't talked or crossed paths in the five months since Gallagher joined the SEC.

After exchanging some pleasantries, Gensler told Gallagher he knew about “your little caper,” as Gallagher later related to others at the SEC. Gallagher responded that he was merely attempting to help put forth responsible regulation.

In the end, Gensler capitulated. He was one of nine officials at both agencies who voted for the rule. Gensler said he would have liked a lower bar, but with so many rules to work on it was significant that “we got it done.”

The energy lobbyists were ecstatic. The NERA consulting firm, which had been hired by BP, Koch, and the other energy companies, boasted on its website about the CFTC's “significant reversal of course.”

Firms including Exelon, ConocoPhillips, AGL Resources Inc., and MidAmerican Energy Holdings Co., some of which had warned investors that their companies might need to register as swap dealers, now reported they would escape the label.

The company that gave the BP Loophole its name wound up registering with the CFTC as a swap dealer after all. Scott Dean, a spokesman for BP, declined to comment. So far, Cargill Inc. is the only other energy or commodity firm to register.

One consequence of the energy firms' win: Banks continue to dominate the business. “The large institutions who have always been major players in the swaps business are probably the only ones who can afford to be dealers because it is costly,” said Sommers, the former CFTC commissioner.

Some energy companies agreed. “This seems directly counter to the goal of Dodd-Frank to increase competition and reduce the concentration in large financial institutions,” said Lance Kotschwar, a lawyer for Gavilon Group LLC, who has worked on the issue for the Commodity Markets Council.

Border Security

Gensler still held out against pressure to erode the CFTC's authority overseas. Few outside his inner circle realized how much of a hand he had in Dodd-Frank's language on the matter.

The provision in the law begins by saying that the new derivatives oversight wouldn't apply to trading outside the country. Then comes the key wording: “unless those activities have a direct and significant connection with activities in, or effect on, commerce of the United States.”

“I know who here drafted it,” Gensler said with a grin, without disclosing the name. “I know exactly who. And I thank them from time to time.”

The banking industry, despite employing hundreds of lobbyists and lawyers to watch the legislation, didn't grasp its significance for almost a year. As the CFTC was writing its policy, lobbyists rushed to delay or kill it, enlisting foreign regulators and U.S. lawmakers in their campaign.

To Gensler and his supporters, there was no sense clamping down on swap trading if the agency couldn't see what U.S. firms did overseas. Didn't AIG, which sold swaps out of a London subsidiary and held a French banking license, prove the danger?

“If you develop a set of rules that are designed to prevent the next AIG and it wouldn't prevent the next AIG, that's a problem,” said Barr, the former Treasury official.

While other countries were developing their own rules, some U.S. officials said they might not be as comprehensive or transparent.

Gensler's education on the issue began much earlier than AIG. Working for Goldman Sachs in the early 1990s, he helped oversee the firm's swaps book in Asia, which he said was recorded on one “massive Lotus spreadsheet.”

As a Treasury assistant secretary in 1998, Gensler was sent on an emergency mission to Greenwich, Connecticut, one Sunday to see if the government could stop the implosion of hedge fund giant Long-Term Capital Management.

After investigating the situation, Gensler rushed back to Washington for the Jewish holiday of Rosh Hashanah, calling Treasury Secretary Robert Rubin from the airport with his conclusion: Nobody had any idea what would happen to the firm's $1 trillion swaps portfolio because it was handled out of a legal entity in the Cayman Islands. Gensler knew well how the industry used such business structures overseas. He had helped set them up at Goldman Sachs.

“I will never forget it,” said Gensler. “This was not a good phone call to make to the secretary of Treasury.”

So when lawmakers were drafting the derivatives bill in October 2009 and considered its overseas reach for five minutes during a seven-hour hearing, Gensler took special notice.

He put his lawyers to work. In late November, the CFTC's Riley sent Agriculture Committee aide Clark Ogilvie and other House staffers four paragraphs for an “extraterritoriality provision,” according to an e-mail.

Wake-Up Call

The language in Riley's e-mail was included verbatim in a 228-page amendment and approved on the House floor without debate. It never left the bill. In late December, after the House passed the measure, Riley sent the language to the Senate. Meanwhile, bank lawyers, including Nazareth, told clients the provision would limit the CFTC, not expand it, according to reports issued by her firm.

“I think that it was fairly common for people at that time to be reading it as a limitation,” Nazareth said in an interview.

Ten months later, Wall Street got its wake-up call. At an October 2010 public meeting at the CFTC, Berkovitz, the general counsel, announced that the law gave the agency “a wide reach and a broad reach” overseas. That could extend the rules to any branch or affiliate of a U.S. bank, even if the branch was selling swaps to non-U.S. customers.

Bankers left the CFTC stunned, according to several participants.

They quickly increased their lobbying. One of their main advocates was Edward Rosen, a partner with Cleary Gottlieb Steen & Hamilton LLP, who became the most frequent CFTC visitor on the issue, records show. Rosen was hired by an informal coalition of about a dozen large banks including Credit Suisse, Citigroup Inc., and Deutsche Bank. Each member kicked in at least $150,000 to start, according to two participants. Rosen didn't respond to requests for comment.

The banks also enlisted allies in Congress and among overseas regulators. JPMorgan lawyer Don Thompson, testifying before a House panel in February 2011 on behalf of Sifma, said some of his firm's overseas clients were threatening to take their business to non-U.S. firms like “Barclays or Credit Suisse or a European bank” that wouldn't be subject to Dodd-Frank and could offer their services more cheaply.

Three months later, Senator Charles Schumer, a New York Democrat with close ties to Wall Street, led 18 members of his state's congressional delegation in signing a letter saying Gensler's proposals could have “significant negative effects on the competitiveness of U.S. institutions.”

Foreign Banks

Later in 2011, the law's co-sponsor Frank weighed in, signing a letter to Gensler saying that “Congress generally limited the territorial scope” of the derivatives law. Frank said he was mainly motivated by concern about conflicts between CFTC and SEC rules.

Foreign banks were focused on the CFTC's reach as well, not wanting certain of their own trades to fall under Dodd-Frank. The Institute of International Bankers, a trade group, led an effort to contact foreign regulators and politicians to put pressure on their American counterparts, according to people familiar with the meetings. In the end more than a dozen foreign regulators complained to U.S. officials about the CFTC.

As the pressure on Gensler increased, he got cover from the Treasury secretary. Geithner, in a June 2011 speech to bankers in Atlanta, said strong U.S. rules overseas would help avoid “a race to the bottom around the world.” He singled out the U.K. for its past “strategy of light-touch regulation to attract business to London, away from New York and Frankfurt” noting it “ended tragically.”

Blowback from the speech didn't help Gensler's case. British Prime Minister David Cameron went to Obama to complain, according to an official briefed on the matter.

In a small victory for the banks in late 2012, Gensler agreed to a six-month delay on determining the rule's reach.

The squeeze on Gensler tightened as the July 12, 2013, deadline loomed. Banks and foreign regulators pressed for a further delay, and Gensler faced an insurrection at his own agency. “No one has ever accused Gary Gensler of being reasonable,” Sommers said at the time.

Gensler hadn't even been able to lock in votes from the other two Democrats. Wetjen gave a speech in London June 25 in support of a slowdown. Chilton called for a compromise.

Then Gensler was summoned to Lew's offices at the Treasury.

The July 3 meeting was an echo of the 2009 meeting that launched his regulatory crusade. Gensler again found himself in a room with the heads of the Treasury and SEC. This time it was Lew and Mary Jo White instead of Geithner and Schapiro.

Lew, who had replaced Geithner in February, wasn't happy to be drawn into the battle, according to the people briefed on the private session, who spoke on condition of anonymity. He said at the meeting that while he didn't want to undermine Gensler's plan, he wanted the CFTC chief to better share information with his European counterparts and the SEC.

After the discussion, Gensler called European negotiators offering to give up some of his demands, a person with knowledge of the talks said. The deal was struck with Europe 10 days later and then internally with Wetjen.

Spokesmen for Lew and White declined to comment.

While details are still being worked out, the compromise means trades involving overseas parties, even if handled by a U.S.-based bank, may fall under rules of another country provided those rules are deemed comparable. The process for making those rulings remains under debate.

The international hole that Gensler wanted to plug remains at least partly open.

The outcome was a relief to banks that had argued Gensler's original vision was too ambitious and would have concentrated too much supervision in an agency ill-equipped to manage it.

Gensler's foreign counterparts were especially angered by his brinksmanship.

Michel Barnier, the EU commissioner responsible for market regulation, said it was unfortunate that the agreement had to come “on the eve of the CFTC's last meeting” on the matter.

The process, Barnier said, “isn't necessarily a model.”

As the CFTC retreats on foreign trading and other fronts, the market that's left for it to oversee is shrinking. Firms are designing contracts so that they fall outside the swaps rules.

The phenomenon has become known as 'futurization.' As far as financial engineering goes, it's simple: Take a swap and call it a future.

The first part of the business to move in that direction has been energy. Large energy companies with total swap dealing near the $8 billion limit were eager for a way to take some of their transactions off the count.

One weekend in October 2012, just before the firms had to start tallying trades under the new rules, ICE, the Atlanta-based exchange, put its futurization plan into effect. On Friday contracts were “swaps.” On Monday they were “futures.”

Within weeks, ICE and CME implemented plans to create similar contracts for the much larger markets in interest-rate and credit swaps.

If enough swaps migrate to futures exchanges, the profitability of alternative trading platforms like Sefs may be less robust. (Bloomberg LP, parent of Bloomberg News, asked a federal court to force the CFTC to act to limit futurization. A judge dismissed the lawsuit, saying that the company had neither the standing nor the facts to support its case.)

About a dozen Sefs have been established so far. Hirani, the derivatives pioneer, decided in the end to hedge his bets, starting a firm that is both a Sef and a swaps exchange.

CFTC 'Slap'

Advocates of less regulation say futurization shows the folly of the government spending several years drawing up complex rules for dynamic markets.

“It is a slap in the CFTC's face,” said Hester Peirce, a senior research fellow at the Mercatus Center at George Mason University, who was on the Republican staff of the Senate Banking Committee when Dodd-Frank was drafted. “It's the agency being told this whole scheme it dreamed up is not going to work.”

CME Group and ICE, along with London-based LCH.Clearnet Group Ltd., are also at the center of another vexing question about the new oversight apparatus. They operate swap clearinghouses. The rules for the first time require them to function as third parties for trillions of dollars in trades moving through their platforms, holding collateral from buyers and sellers in case a transaction goes bad.

The benefit is that clearinghouses will become central hubs where a bank's exposure to derivatives can be monitored and regulators can get information on the market.

In the aggregate, though, a clearinghouse also winds up holding all of the risk because it acts as the other side of the trade for both buyers and sellers.

If a market blows up, the clearinghouse could turn into a sort of supercharged AIG, unable to cover the losses. That could create “potential vectors for the transmission of systemic risk,” the Clearing House Association, a trade group representing the largest commercial banks, said in a December 2012 report. Banks have been warning that clearinghouses aren't holding enough capital and are using less-liquid collateral, such as corporate bonds rather than cash.

Duffy, of CME Group, said the exchange has put protections in place. “I would have concerns, too, if my trades were in a clearinghouse that I did not know exactly how was being risk-managed,” Duffy said. “We are working with the dealers to make sure they're comfortable.”

Gensler, too, has expressed concern about clearinghouses. In May 2010 he wrote to Lincoln and Senator Christopher Dodd, a Connecticut Democrat who co-authored the financial regulatory law, in a failed attempt to keep the firms from being put under Federal Reserve supervision. Access to the central bank's emergency lending could raise “the risk of clearinghouse failure and the possible need for a future bailout,” he wrote.

Gensler said that having central clearing is still better than entrusting the system to individual banks.

Analyzing whether the CFTC's new rules will make the financial system safer than it was in 2008 remains an exercise in estimates. Derivative trades are still largely private, and records kept by clearinghouses and other information repositories aren't standardized. A review of the best available data from those sources and government filings shows that CFTC regulations for trading, clearing, and reporting about transactions may be felt in less than a sixth of the market.

Calculating Market

The total value of OTC derivatives traded worldwide is $633 trillion, according to the Basel, Switzerland-based Bank for International Settlements. More than half of that is held outside the U.S., according to government records, and could be mostly excluded from CFTC oversight.

Of the $300 trillion in derivatives held by U.S. banks, only a portion would fall under all the new restrictions, according to the review. The biggest banks sometimes trade half their swaps with overseas clients, and under the cross-border policies, those deals might be subject to foreign law instead of Dodd-Frank. After subtracting trades that fall under the foreign-exchange and end-user exemptions, as little as $100 trillion of the total $633 trillion could feel the full force of Gensler's rules.

Jeffrey Harris, a former CFTC chief economist, said the exemptions mean that “a very small fraction of the total market” will fall under the new requirements.

“It may be as large as $100 trillion, but at the current time it is probably substantially smaller than that,” he said.

Gensler said that such a calculation doesn't fully capture the scope of the CFTC's new authority, which is laid out in 60 new regulations that touch on a broad swath of derivatives held in U.S. banks, a market worth $300 trillion. Swap dealers have already reported trillions of dollars in transactions that wouldn't have been public under the old system, he said.

“The market is shifting,” Gensler said. “The public is far better off today with the transparency and the reforms we put in place.”

As the CFTC prepares for operational oversight, it faces more challenges. No longer a backwater, the agency's resources haven't caught up with its new responsibilities. This year it has a $207 million budget and 700 employees, making it less than a fifth of the size of the SEC.

It is “completely wrong” for Congress to keep the CFTC's funding so low, said Schapiro, who as SEC chairman had a $1.4 billion budget at her disposal. If that isn't fixed, she said, “We will have a false sense of security about what these rules can do for the stability of the financial system.”

Chairman's Future

For Gensler himself, the future is also undetermined. With his tenure required to end by Dec. 31, Obama has given no sign that he will be reappointed to the commission, according to people briefed on the matter.

Investor advocates see that outcome as a measure of his success. Gensler shows that “you can go through the revolving door and serve with independence and integrity,” said Jeff Connaughton, a former Democratic Senate staff member who was involved in the Dodd-Frank debate and last year wrote a book chronicling the rise of Wall Street's lobbying machine.

Hirschmann, of the Chamber of Commerce, said that while Gensler has dealt in a straightforward way with his trade group, he overreached on some proposals. He said it's unclear whether the rules will accomplish Gensler's goals.

“My guess is we will find some of it works, some of it didn't, and we will have to come back and fix it,” Hirschmann said.

Gensler said he understood from the beginning that the original vision wouldn't survive intact.

“Along the way that means there is some moderation and some compromise,” he said.

He and his inner circle wanted to focus on the most important measures and were able to “maintain the core,” he said. Even with reduced scope, the accomplishment is enormous, he said.

“This is a $300 trillion market coming out of darkness,” Gensler said.

Copyright 2018 Bloomberg. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.