Best Tools to Work with Chess PGN Files

Today I will give a brief overview of how well different Chess Database software packages support the PGN file format.

Fritz - reads and writes

ChessBase (multiple versions on various platforms, including Light/Full, Windows/Mobile, etc) - reads and writes, except for CB Light

Chess Assistant (Full and Light) - read and writes
ChessX - reads, but writes have some issues
SCID - reads, but does not write

(If you know of another software that supports PGN files - please leave a comment.)

With so many tools that support PGN files, does that mean that you can use them interchangeably to update the same files? Yes and no. The major caveats are as follows:

• Each tool exports moves with a slightly different new line format
• Each tool (or even different versions of the same software) updates headers in its own way, putting PGN header tags in a different order. That results in issues such as player names in the game headers cannot read properly by some versions of the same packages. 
• Additional formatting, such as arrows, diagrams, highlighted squares, and engine evaluations also follow a different convention. So arrows you draw in Fritz, will likely show up as garbage if you open up the same PGN file in Chess Assistant and vice versa. What's even worse - if you add a diagram for a position in Chess Assistant, hoping to be able to print it from Fritz, you'd be in for a disappointment.

To learn more about this (and to possibly bypass some of these issues) - read my recent post Editing Pgn Files without ChessBase or Other Database Program – Syntax Highlighting or other posts I wrote about PGN.

Editing Pgn Files without ChessBase or Other Database Program – Syntax Highlighting

Six years ago I already once wrote about pgn file format, and the benefits of using it for storing and sharing your chess data. While working with many pgn files, I want to be able to load them into a text editor, and do various operations on it that can’t be done easily in a chess database, such as SCID, ChessBase, Chess Assistant, etc. Notepad++ , my favourite editor, lets me

• check spelling
• translate from one language to another
• do massive search and replacements across the entire file (or even many files)
• and many many other things you would normally do in a powerful text editor

The problem with text editors is that normally pgn like any text looks Black/White, and is not very pretty.

Instead I want it to look like this:

Here is the trick: once you loaded the pgn file, make sure to set language to “Pascal”, so that you get the proper syntax highlighting for:

• game headers
• different types of braces (for annotations and variations)
• move numbers

I never programmed (and probably never will) in Pascal, but this nice little similarity between syntaxes between PGN and Pascal comes really handy!

Chess Position Trainer – test your opening knowledge!

I have written a fair bit about how you’d want to be careful about choosing the openings that you play and opening preparation in general. But how do you actually remember the multitude of lines that constitute your repertoire? If you carefully analyzed certain openings, and your repertoire consists of your original ideas – memorization likely won't be an issue. But if you never played a line before, and barely can think of a couple of plans – as soon as a position on move 10 comes up on the board, you'll realize pretty soon that:

• The devil is in the details – understanding positional ideas is great, but you still need to play specific moves to implement those ideas
• Move order matters
• some positions look similar, but the move you remembered and thought should be played in this particular position is actually good in a different sub-variation, but is inappropriate in this particular case

To test and refine my understanding of an opening – I usually play blitz games on ICC, look at them later and see whether I deviated from my own repertoire, or whether there is actually a gap in my repertoire that this game revealed. While practice games are super-useful, that type of training has some limitations:

• you have no control over what opponents will throw at you (you really did not want to test your knowledge of 1…b6 !)
• in the rush of a blitz game - you may not focus too much on playing the move that you had prepared (I already spent 30 seconds on the opening, I’d better move something quickly)
• once the game has ended - you also need to have the stamina to go back and compare your moves against what you had planned in the preparation

So there is a need for a special tool to test your opening knowledge! I have always admired the developer who put up a ton of effort into creating a free tool that lets you manage your opening repertoire and then practice it in a training mode – Chess Position Trainer. It always felt like a tool I would want to write myself if I had the skills and time. As it often happens with free software - the author had to fill the gap that no other free or commercial application would cover sufficiently for him.

However I have not used it for a while as it seemed that all my data was already in pgn and chessbase format, and my opening repertoire existed outside of Chess Position Trainer ecosystem. What was even worse – my repertoire had too many gaps (it still does), so I spent the last year going through books, databases and my old games, deciding what I should play against such and such line. Today I imported the opening repertoire I had created with Chessbase Light into Chess position Trainer, and here we go – I can practice my openings again!

In the training mode – the UI looks like this:

Training mode

A few points I like about its training mode:

• games are thrown at you relatively randomly
• Your progress is tracked (I have not dug into the details)
• The board is large and I like its simple look – often underestimated but very important, but some $100 packages got that part wrong.
• You’re told immediately whether you made a correct move (“correct” means – the move you yourself defined to be in your repertoire)

Sometimes it keeps throwing the same line at you until you actually know enter all the correct moves:

Training Mode is a pretty fundamental part of Chess Position Trainer – you can tell that from looking at training options (that’s a lot of Tabs!):

I would agree with what Stefan (the developer) wrote in this blog post – "this time I have already at hand a chess tool to study my material."

Outside of the Opening Training mode – the UI looks like this. This is one of the most well designed user interfaces for a chess program I’ve seen so far. It sounds silly, but buttons are large and convenient to press. It feels that the author actually uses this program himself, so he tried to make things easy for himself – and therefore, for any other chess player using his product. At the core of the software - the focus is as much on positions as it is on the moves, which makes sense for opening preparation.

Cafechess control has been updated!

If you are a chess fan, who is also into programming, and in particular use the Microsoft technologies, you might be glad to learn that the author of Cafechess is continuing development on it. He posted a new version on CodePlex, and the demo application looks a bit different from the 2004 version that I had seen before. This control has been used in ChessPositionTrainer, which is a pretty good tool, especially considering it’s free. I’ve also used some code (pgn parser) from this project for posting some of the games and diagrams. I talked about it in my other post in more details.

Database files for chess books

Having recently flipped through all 7 of Kasparov's recent books (5 volumes of My Great Predecessors, Revolution in the 70s and the latest Kasparov-Karpov 1975- 1985), I want to have the games in a computer database. What a common nerdy desire! Books are great for calmly re-playing through the moves, but what if I want to add a variation refuting Kasparov's analysis (notes on the margin?), or reference a game from the book in my own analyzed game? I also want to have a collection of games that are useful for understanding the middlegames arising from the openings that I play, and Kasparov's books are good for learning about historic development of plans and ideas (but only some games are relevant to my repertoire).

Seems like I already paid for the contents of the books, I would not be violating anything by having them on my computer (especially - if they are without annotations). A while back gambitchess.com started a great collection of database files for published books, but they don't have Kasparov's books. Yet looking around the Internet, I can only find 2nd and 4th volume of My Great Predecessors in pgn/chessbase formats. Chessgames.com has the collections of games that I need, but I can't download them in one shot without becoming a paid member. Seems like I might just have to download them one by one ...

Chess database formats - PGN vs. Chessbase

In what format do you store your chess data? This question has plagued me for years, so I finally decided to blurb it all out. There are at least popular 3 file formats I can think of - Chessbase (CB) binary format (cbh, cbv, etc), Chess Assistant (CA) binary format, and standard text pgn format (with the spec here). Most other chess databases, such as SCID, Jose, etc also have their own binary formats, but I am not as familiar with those. There have also been a few efforts to represent chess databases in some open XML format, but of all schemas proposed, none has really gained enough popularity. At least most chess games available on the net are still available only in either PGN, Chessbase, or CA formats.

So what tools are available for dealing with each format? Virtually every chess tool can read pgn. Chessbase does not handle Chess Assistant at all, Chess Assistant allows you to read Chessbase format, which is pretty impressive, given that no other non-chessbase tool does that. So what are the advantages pgn format versus say, chessbase?

PGN advantages: 1) it is free to in the sense that you don't need a proprietary software to view its contents (SCID would do the job), has an open spec, so 'anybody' can implement a parser. There are few parsers available in source code (I managed to find c++, C# and perl parsers, there are probably others). That being said, I suggest you do not write your own parser, because it is not a trivial task, but rather take an existing one (of course as long as license is not an issue).

2) since PGN is text-based - one can just load them in notepad, as long as the file size is not prohibitively large. Pgn is also very readable - the moves are just in algebraic notation. If you store your personal games in PGN, you can version control them as well, and look at the differences between revisions. It is also trivial to merge multiple PGN files into one via a one line in DOS.

3) A lot of free command line and GUI PGN tools are already available. pgn-extract is a great command line tool for filtering pgn games for material, position, or tag information (e.g. ECO, or Players). Palview is great for generating html with javascript for replaying games. In fact I use pgn-extract and Palview together to generate content for this blog, but this is worth another post. This site is a great resource on chess database utilities, most of which operate on pgn.

So if PGN has so many advantages, why use anything else for storing chess databases? Why create all these compatibility issues between multiple chess database vendors? Same reason as why XML is not used as a backend storage for storing data in SQL databases - performance.

Chessbase format provides more than performance improvements though: 1) In addition to performance, filesize is also smaller - chessbase splits up databases into multiple files, and that allows for some normalization of headers, etc.

2) Multimedia support - one can embed audio and video into games.

3) it can be a real database - in addition to raw moves, chessbase stores other metadata, such as opening and endgame keys, allows to tag positions with so called "medals" and so on.

I have been using Chessbase light for storing my games since around 1998, and just don't want to lose all the tagging that I've added to my games for many years. So currently I maintain keep old databases in Chessbase format, and whenever I create new smaller databases (say, for selected games by Kramnik in the Sveshnikov), because Chessbase light can still edit them, performance is not an issue, and I don't have to re-export all games into PGN for using Palview.