Two hard days and what comes next

I’m Johannes Grof, a student at HTBLA Kaindorf, and I build ÖffiGo on my own, alongside school. Behind the app there is a server, a database, several data providers, a status page and this website. Keeping all of that running is more work than it looks from the outside. When something breaks, I often notice it in the break between two lessons or in the evening.

I do my best to keep the app and the server stable. On Wednesday a lot went wrong at once anyway, and that is what this post is about.

What happened

All times are Austrian time.

• Tuesday 20 October, evening
  Railway, the platform my server runs on, could not deploy new versions for a few hours. The app kept working, an update just arrived later.
• Wednesday 21 October, from 9:24
  Railway had a major outage and stopped building new versions in all regions from around midday. Whatever was already running kept running, ÖffiGo included.
• Wednesday, 12:00 to 15:00
  A data provider’s timetable interface answered some requests with an error, in bursts: three errors in ten seconds, then quiet for minutes. Departures and connections often only loaded on the second try.
• Wednesday, from 15:59
  No live data came in for the Vienna U-Bahn, trams and buses. ÖffiGo showed the scheduled times there instead. The data was back on Thursday at 6:37.
• Wednesday, 17:39 to 20:40
  The timetable interface was down completely for short-notice maintenance. Departures, arrivals, trip details, journey planning and ÖffiGo in your assistant did not work during that time. Search, disruption notices, punctuality and this website kept working.
• Thursday, 0:10
  Railway reported a security incident and advised every customer to replace the access keys stored with them.
• Thursday, all day
  Railway is bringing builds back step by step, larger plans first. My plan isn’t back yet, so I still can’t ship anything new. Railway’s status page has the current state.

Behind the scenes

When the first errors came in, I wanted to be sure they were not coming from my side. So I queried the interface directly, without ÖffiGo in between: 48 requests in five minutes, two of them failed. The problem was with the data provider.

While doing that I noticed that ÖffiGo handles bursts like these badly. After four errors in a row, the server left the interface alone for a while so as not to add load. That is well meant. With three short hiccups, though, it meant no departures for anyone for a stretch of time.

I wrote and tested the fix that same Wednesday. I could not ship it, because Railway wasn’t building anything on exactly that day.

For me that was the most frustrating part: the fix was ready, and all I could do was wait.

After the security notice I checked the database for outside access and found none. New access keys for every service are ready. They take effect with the next version Railway accepts.

What that meant for you

Three hours without departures is about the worst thing that can happen to a public transport app. If you were waiting for a bus on Wednesday evening, you had to look somewhere else. I’m sorry about that.

What went well: status.oeffigo.app showed the whole time what was not working and since when. You can also get updates from there by email.

What I am rebuilding

That same Wednesday, PHP2Shell became public: two critical flaws in PHP, the language a large share of all websites runs on. Within the day there were millions of attack attempts worldwide, and Germany’s federal security agency issued a warning. ÖffiGo doesn’t use PHP and wasn’t affected. Together with what happened at Railway, though, it tells me that attacks and outages at services you build on are no longer the exception. I can’t prevent them. I can build ÖffiGo so it gets through them, and that is what I’m working on:

• A circuit breaker per service. The server no longer shuts an interface off after four errors in a row. It looks at the error rate over the last minute, for each service separately, and a fast error gets a second try straight away. Three hiccups in ten seconds now cost at most three requests. Built, goes live as soon as Railway deploys again.
• Measuring before you notice. Since Wednesday, checks in Frankfurt and Zurich test from outside whether ÖffiGo answers. Next, they will message my phone right away when something is wrong, and the server will measure every outside connection itself: error rate, response time and how fresh the data is.
• The last known board when nothing new comes in. If the timetable interface fails, the app should show the last departure board and say how old it is. Planned.
• A second way to the data. A second, direct source for live data in Vienna, and for emergencies a timetable ÖffiGo can show departures from even without the interface. Planned.

Where I want ÖffiGo to go

My goal is a public transport app that is fast and keeps working when a service somewhere fails. For that, ÖffiGo should stand on several legs: several data sources, its own measurements and a server that expects outages.

The data providers stay important. Without them there would be no ÖffiGo, and I’m grateful they support a school project. Independent, for me, means that ÖffiGo still shows you something useful when one of them is gone for a while.

If you notice something or have questions, write to me at contact@oeffigo.app. I read every message.

Thanks for using ÖffiGo.

Johannes