Steam OS games aren’t quite where they need to be yet, it seems, to compete with their Windows counterparts, according to a report from Ars Technica.

The site tested the Valve and Microsoft’s platforms with some standard benchmark tests as well as some one-to-one game and hardware comparisons, with Windows coming out on top to various degrees both times.

First, they loaded up Geekbench 3 in both operating systems in a dual-boot setup on the same hardware. In that case, Windows 10 came out on top, but Steam OS, which is Linux-based, was “within the same order of power magnitude.”

Getting a game running, though, showed some more pronounced differences.

To do some live-fire tests, the site selected a couple games that have full Steam OS ports: Middle-Earth: Shadow of Mordor and Metro: Last Light Redux. They also ran tests with each of Valve’s Source Engine games – Portal, Team Fortress 2,  Left 4 Dead 2, and Dota 2.

In all six cases, there was a pronounced difference in performance. For the first two games, tests were run at a variety of different settings. The differences came out with Windows displaying anywhere from 25 to 55 percent more frames per second than the Steam OS configuration. The Source Engine games, which should really put Steam OS on display, were ran at one average setting level and in most cases showed pronounced differences of at least 10 frames per second up to nearly 40 frames per second improvement in Windows 10 versus Steam OS. Left 4 Dead, it should be noted, was nearly identical on both.

An important note made by Ars Technica is that games natively built for Windows are more likely to show this. If a game is built for Linux and OpenGL, the performance difference should disappear. While this test is nowhere near comprehensive – it doesn’t test different graphics cards or configurations and only a few games were tested – it, I think, gives a pretty good picture of the experience the average gamer can expect when trying to run popular games on the system.

Steam OS is in its infancy, while Windows is an overwhelmingly common operating system with mature drivers and an API used by most games, so this isn’t surprising. Operating systems, hardware drivers, and PC games are all incredibly complex, and the latter two are built with a specific operating system in mind that captures as much of its potential audience as possible.

If Valve can keep the Steam OS audience growing and show them as an audience eager to spend money on games made with Steam OS in mind, then game companies are going to start keeping that in mind as they develop games while Nvidia and AMD will begin to see value in optimizing drivers for the operating system. Until then, though, it seems as if moving to Steam OS comes with sacrifices not just in the available list of games but in performance on many popular games as well.