IST and Sportal.com: Live on the Internet!
September 14, 2001

by Claire Sponheim

No wonder live broadcasting is so important for the sports dot-coms, their sponsors, and their advertisers. Vast legions of sports fans trapped behind firewalls, in offices all over the world. And all with access to the Internet.

If you're a sports fan, you know that live means now. Not 30, 60, or 90 seconds delayed. And it means not getting bumped off your live feed as the site gets swamped with hits.

Sportal.com -- one of the largest international online sports portals -- won the franchise for the official site of the Euro2000 soccer tournament. The site would broadcast live text commentary and scores for all the games in five European languages.

This is the story of how U.K.-based Sportal worked with Imperial Software Technology (IST) to design, build, and manage this scalable, real-time match-tracking service.

The Size of the Task

Whichever way Sportal did its calculations, broadcasting the European Soccer Championships to fans across the world was going to require scalable new technology. The company was going to deliver real-time sports commentary and results to very large numbers of people.

Two years earlier, the official site of the World Cup in France broke records with over 80 million visitors during the competition. Euro2000 was likely at least to double that figure. If only one percent of visitors decided to follow the game live, more than 50,000 would be online for each of the 31 matches in the tournament. Also, probably many more fans would log on during the final stages of the competition.

The Initial Requirement

In October 1999, Sportal approached IST to produce a Java[tm] applet front end for a system that would broadcast sports scores and commentary in real time over the Internet.

IST, founded in 1982 as a spin-off from Imperial College in London, was strong in both user interfaces and the Java programming language, having first developed X-Designer, then Visaj, a Java visual application builder.

Soon the scope of the work grew to meet the real challenge. The result was the following: DeltaStream Web infrastructure for scalable, fast distribution of content to applets on the desktop; Web-based servers for inputting editorial and statistical content; and the Java front end, a flexible multiapplet design that could be customized by graphics designers, not programmers.

Traditional HTML refresh polling -- with up to 30 seconds delay -- is not real-time. It's not even close. Sportal's no-compromise requirement was for continuous connections to every desktop -- no pauses, no dependence on network traffic, no transmission breaks. Once customers connected to the Java applet front end, they would remain connected. And that meant a system that could easily maintain continuous connections across the Internet to half a million users.

System Design: Building Task-Specific Servers

The DeltaStream system has two types of servers -- the input server (also known as the pump), for collecting and publishing information, and the distribution server, for relaying and distributing content to users. Says Andy Bartlett, IST's principal technical officer, "What we did was build technology in both the Java programming language and C, and we used Web servers as the model. Everything used HTTP and the Internet: entering editorial content, forwarding data to the main public-facing servers across the world, and driving the Java applet front end. We looked at off-the-shelf Web servers; for example, I had a good look at Apache. But it was quite large, built to do lots of general-purpose things. We wanted it to do one specific thing. Our code was focused on handling large numbers of users and maintaining those connections. All the design decisions were focused on that."

The Pump: Collecting and Publishing

The pump collected information from sports editors inputting data across Europe, and from match statistics feeds distributed in XML. This server was written in the Java programming language because of its flexibility and extensibility. This server also published Web pages, but its main task was to publish real-time data into the DeltaStream, the continuous XML stream that was picked up and relayed by the distribution servers.

Once a game started, editors from all countries would input their commentary. Their browsers connected to the pump and posted back the data from HTML forms. Says Bartlett, "Like content-management systems such as Vignette, we had our templates and our language dictionaries, which generated forms on the fly. These appeared on an editor's screen in the appropriate language -- Danish, French, Italian, Spanish, and so on."

Up to 20 editors connected to the pump during a game. Since the pump was a Web server, the editors could broadcast their commentary to the world, using laptops or even wireless modems, as long as they could get an Internet connection. And the editors didn't have to install or use any software; their interface ran in their favorite browser using standard HTML and JavaScript[tm] technology.

The Distribution Servers

Says Barlett, "We needed something very powerful in front of the pump to handle 100,000 simultaneous users. It would have to take the data stream and distribute it in a way that could be scaled easily and cheaply."

The solution was another specialized Web server that was specifically designed to serve and maintain very large numbers of connections. It sat in front of the pump, took the feed, and redistributed it to Java applets in browsers. It also made sure the system never got blocked, no matter how many new users attempted to connect. Once a user was in, the user received the data feed, and nothing ran interference on this. New users connected in an orderly fashion, just like getting into a stadium through the turnstiles.

The distribution server was a C program, with a small footprint of just over 50KB. It was about one-sixth the size of the Apache Web Server and approximately one order of magnitude faster because of its singular focus. It acted like an amplifier, increasing the system's capacity. Furthermore, distribution servers could connect in parallel or series, so the more capacity needed, the more machines running distribution servers were brought online.

Figure 1 shows the layout of the inputting commentators, the pump, and the distribution servers.

Figure 1: IST/Sportal.com Distributed Architecture (Click image to enlarge.)

Rack Space and the Euro2000 Hardware Architecture

Live broadcasting was only one component in an architecture that was designed to serve 150 million visitors in a three-week period. The DeltaStream servers were hosted on Netra[tm] servers because Sun servers were chosen to host the whole operation. One major factor in the choice was rack space.

"This was not long after the 220R, 420R, and Netra[tm] T1 model 105 line of boxes became widespread," recalls Matt Clark, at-the-time technical architect at Sportal. "Previously, to get four CPUs and some useful RAM in a box, you had to use an E450 [server], which is huge. But now it all fit in 4U of rack space. [Note: U = 1.75 inches.] Additionally, because those boxes are designed for data centers, they have all sorts of cute stuff like Lights-Out Management, which is very important for a globally distributed system."

"As for why we selected Sun hardware," Clark explains "Sun was Sportal's primary platform anyway, and as it happened the newer rack-specific range was perfect for us in Euro2000, where there were quite strict space constraints in certain locations. Above and beyond space reasons, we just knew that if everything was set up right, the Solaris[tm] Operating Environment and Sun servers would never go down -- or so we thought. It turned out that the biggest problem we had was the well-known cache coherency issue on the E420R servers, causing a few of them to keel over at random a couple of times."

Sun E420R servers were used for the back-end heavy lifting, with 4x400Mhz CPUs and 4GB of RAM, with external Sun StorEdge SRC/P Intelligent SCSI Raid Controller[tm]. Netra T1 servers, with 1GB RAM, 1x400MHz CPU, and 2x18GB disks were used for the front-end work (Web servers, log aggregators, DeltaStream live match servers, and local management). Even at peak load, the site was published off just two Netra T1s.

There were two publishing centers -- L.A. and London. They published to four major and six minor cache farms, totalling 28 CacheFlow caches. Each farm (or point of presence, PoP) had a Netra server as a log aggregator, and six had another Netra server, which hosted a public-facing DeltaStream distribution server. ArrowPoint load balancers were used for both global and local load balancing.

And even "static" Web-page distribution used push technology. Clark tells how the company had to switch at the last minute:

"We had originally pursued a pull model, where caches periodically checked for new content, with a push model in reserve just in case. We moved to the push model just a week before the tournament, when the "Get-If-Modified" requests from the caches were sending the Web servers to a load average of 20. The reason was that every page on the 17,000-page, 2GB site was being visited at least once every half hour! Still, the Web servers never fell over and were always manageable. And we were able to execute a smooth change to the push model, after which the load was never greater than one or two, even when the system was delivering a sustained 100Mbps of traffic."

The DeltaStream Netra Servers

User-facing distribution servers were hosted on geographically dispersed dedicated Netra servers that connected back to the main distribution server in London. Says Bartlett, "We used the Internet itself to lower the bandwidth requirements. Rather than require vast amounts of bandwidth in London, this main distribution server connected to other distributed servers in Los Angeles, Virginia, and Amsterdam." Users were routed to whichever server had bandwidth. A benefit of being global was that during an afternoon game in Holland, most people in L.A. were asleep [resulting in light traffic]. Typically, in a game with 100,000 users connected, each server machine would take about 20,000 connections spread across the three local distribution servers.

Fault Tolerance

The distributed-server model is fault tolerant. If the pump goes down, it can be restarted and will pick up exactly where it left off. If a distribution server's connection is broken, it reconnects to the stream. If a user's browser loses its connection, it requests a reconnection from the last data item it received. The system starts sending data from that point forward. Remarks Bartlett, "We wanted to make sure we didn't swamp people with data. They would reconnect and get only the data they had lost, then just pick up from there and carry on."

The User's View

The Java applets connected through hardware routers to one of the distribution servers. If for any reason they could not connect or lost their connection, a little stoplight on a user's display turned red. Once the applet reconnected, the stoplight turned green and everything continued. Before this simple device was introduced, any failure (or quiet moments) resulted in frantic reloads by thousands of users. Because the stoplight made disconnects visible, users were not forever pressing the reload button when the game was quiet and no data was sent.

The View from Rotterdam

"The tournament went much as planned," says Clark, "in that for the most part the system ran itself." He recalls the live broadcasts: "During matches, of course, we crowded round the traffic graphs -- peak over 5 seconds = 1Gbps; peak over 5 minutes = 100Mbps -- and marveled at the number of people: tens of thousands logging in to the DeltaStream match tracker. There was great competition among the editors to be the "caller" for the games, whose commentary went out live all over the world. Luckily, the reliability of the systems was such that for some matches more than half the technical team was at the game rather than back at base. We had a rule that one person had to be within ten minutes of the hotel at any time, but that was it for cover."

Half-Time Debugging

Reliability simply means that the system doesn't fail. Often this comes from a good design and forward planning. But sometimes a running system needs to be fixed, even if it hasn't broken yet.

Bartlett recalls one such example. The distribution server had several interfaces: HTTP requests from a user, images or data stream sent from the server back to the user, and the back-end connection to the pump. All interfaces were designed to behave in a non-blocking way. But there was one unforeseen case that was blocking only occasionally.

In the early parts of the tournament, all the support team could do was keep track of the bug. When they noticed it blocking, they would simply restart the server. Being fault tolerant, this was not a serious problem, even though it meant that the system required continuous monitoring just in case. Says Bartlett, "But we didn't want the situation to continue beyond the quarter finals, because as these things go, we knew it would block just as the winning goal was scored in the final! I had a good idea where the bug was but couldn't replicate it."

The tournament phase was approaching, and it was during one of the last round-robin games -- with about 20,000 people connected -- when the bug appeared again. As soon as the whistle was blown for halftime [lasting about five minutes], Bartlett switched in the test version of the distribution server containing diagnostics and waited. Unaware, all users were now attached to and being monitored by the test system.

Recalls Bartlett, "With about 30 seconds to go before kickoff, the problem showed itself. We had the diagnostic trace, which found the bug, then we switched out the test system and switched in the live system, and no one was the wiser. The bug turned out to be one particular [older] browser making a non-standard HTTP request. We made the fix overnight."

Bartlett continues, "The stress of debugging in real time is amazing. Everything seems to happen in slow motion. It's horrible. You're watching your software uploading onto the machine, and your heart is beating loudly, and you know that in 30 minutes you've got to give the green light to the commentators all across Europe to start. It's a very, very high- pressure situation."

Concludes Bartlett, "This is what it must have been like in the early live days of television. You learn what it means to broadcast live Internet content. Most Internet sites are fairly static and passive." But in this case, a form of soft multicasting of live-content packets flows across the Internet. Says Neil Smyth, IST's director of Business Development, "It's the difference between reading the newspaper and watching breaking news on TV."

In Conclusion

Smyth sums up the project, "We wrote a distributed application that was embedded in the Internet, using the Internet as our infrastructure. The Java technology-based Web servers were not used for serving data to end users. They became components in an embedded Internet architecture. Specific Web servers were designed to do specific jobs. Also, writing a Web server is not necessarily about writing a replacement for Apache. You simply have a data flow that you want to manage. The Internet takes care of the flow, and the Web server is transformed into a powerful tool for handling the data."

The Overall Message

"The message here is that the Web lets you do so much more than publish Web pages," concludes Bartlett. "It's powerful for getting programs to work together and pushing data around. The Internet is a massive off-the-shelf communications resource, ideal for writing scalable, fault-tolerant distributed applications and lightweight thin-client user interfaces."