Looking for a Large Chess Database

With a number of websites offering online chess databases proliferating (the Chessbase one looks nice), it is still surprisingly difficult to find a free large collection of chess games for download – at least from a legitimate source. Ok, not difficult – impossible. With the ICOFY project being suspended, my options are limited again to downloading TWIC updates and merging them to my existing older ICOFY database.

If anyone is aware of any new source of a relatively consistent free chess database for offline download in PGN, SCID or Chessbase formats – please post a comment or let me know!

Checklist for Maintaining Opening Repertoire in Digital Form

To improve your chess preparation, you want to keep track of your openings in a digital opening repertoire. I already blogged with an example for creating an opening repertoire in my post titled "Opening Preparation - How to make a Tree". It can be maintained in various database  packages, with the output being stored as a single game, looking like this (example from SCID):

Now I am reviewing my opening preparation and I came up with a checklist for maintaining such a tree:

1. Add diagrams to have at least one diagram per game
2. Cut down lines that I will never play myself
3. Add verbal evaluations and explanations of plans wherever possible
4. Add symbol evaluations wherever possible
5. Add my slow and rapid games for completeness
6. Re-evaluate sharp lines with computer and give them a more definite evaluation
7. Remove lines from the repertoire that I completely stopped playing and will never play again
8. Keep repertoire games sorted by ECO
9. Search and highlight tactical combinations anywhere in the repertoire database
10. Label each repertoire game appropriately to distinguish my White from my Black repertoire

This may seem tedious, but this sure beats maintaining the same information on paper!

SCID vs PC latest release – docking feature

I am very impressed with the very latest release of SCID vs PC 4.9. It has a docking feature, which up to this point had been missing in this otherwise excellent (and free!) software product.

I can move sub-windows around, and as one would expect, as I navigate through the move in the game, SCID updates the tree view.

As a side note, while playing with docking, I also realized that any database in SCID can also be opened as a tree, which is very handy, and allows to look at multiple trees at the same time!

SCID Score Graph feature

SCID (also available as “SCID vs. PC”) has a great feature where based on annotated games (that SCID can also do for you) – you can generate evaluation graphs. They are called Score Graph and are available under tools menu.

Clicking on the place in the graph where the evaluation took a dive on move 14, I can find that White just dropped a piece by leaving the bishop unprotected.

 

In this case this gives me a better picture of the flow of the game than many verbal annotations that often accompany chess games. SCID does it, and not only for the game that you just played, but for any game stored in your database with engine evaluations!

SCID Feature – Personal Chess Rating Graph

I have always been wondering why many commercial chess programs don’t let me visualize my rating progress from my own database of games, and was glad to find that free chess database software SCID has this feature. Here are a couple of my charts:

My ICC rating for games with “standard” time controls (15 min+):

The drops usually correspond to times when I had to re-start my ICC account and gradually grow the rating from scratch.

My CFC/FIDE rating:

These graphs not only shows periods of rating rise and fall, but also periods of declined activity, not bad at all for a free program! My CFC rating clearly shows that most of my chess growth happened in 1998-2001. That time range coincides to lot of data points, confirming that to improve – you need to play more!

Solving Simple Chess Tactics – Part 1

To improve your tactical vision, there is no need to buy expensive chess software, it can be done with free copy of ChessBase Light and database that can be downloaded from this page – the one under “Over 3500 tactical training positions from actual games”.

In the video I then follow these steps:

1. Load a game and change board to use “Training mode”
2. guess which move should be played, hit “forward” key
3. If my guess is correct, mark the game in the database as “deleted”
4. Load next game with “F10” shortcut

Here are few simple positions for you to solve. It is White to move in all of them.

     

     

The video shows the setup and solutions:

Fritz 13 Let’s Check Feature

Fritz 13 is adding a new feature that is called “Let’s check”. The feature allows to share engine analysis with other users and upload it to the “cloud”. Such analysis would be easy to look at in one of the Fritz panels:

The benefits of this kind of feature seem very exciting:

• it will allow fast access to all previously made engine analysis
• reduce the need to redundantly run engine on positions that someone else has analysed
• encourage sharing between chess players on an unprecedented scale
• it actually has a UI that is easy to understand. Convekta’s IDea still seems very complicated to me when I read explanations of how it works

The Video tutorials are brief and to the point:

But the scary aspects of the feature seem a lot more obvious:

• spying on each other – sounds like the option is on by default!
• in perspective, this takes us much closer to chess being completely solved
• ChessBase may control data contributed by many chess players, many engines and so on. While games are now being shared in databases produced by more than one vendor, ChessBase having billions of extra positions stored in their private databases will give them a monopoly over most of chess data, data contributed by their own users, who would now have to pay yearly membership fees to access that data.
• focus is on engine analysis, although I think this kind of system should have put emphasis on people’s verbal commentaries (Comments Network feature does seem to go in that direction though, although why not instead use existing GM comments that are spread out through their MegaBase already?)

Yes, this is just a tool to help players with what they do – use best engines to solve mysteries of various chess positions. Some would argue that this is inevitable anyway, but I find this tool more disturbing than anything else.

The biggest question I have though – for how long are they going to have enough storage space to maintain trillions of possible chess positions??

SQL to Chess – Querying Chess Database

Most Chess Databases use their own format for storing information about chess games and the games themselves. If you want to query or extract any data out of the database package – your best bet is to use pgn files, although recently Aquarium also added ability to export data via scripts. In either case – performance is likely to be an issue as text files can get pretty large. When I learned that Jose Chess uses MySQL, I was curious to find out how to connect to its database so that I can search it with my own queries. Eventually, I was able to get it to work, and run a query like this against its large database which comes with Jose Chess and contains about 1.5 million games
 

This query shows the opening that scores the best for White – A94 - who would have guessed! The most popular index is B22 – 2. c3 Sicilian. There is obviously now opportunity for running more metrics on players, openings, openings trends, etc.
Getting all the data extracted out of Jose Chess obviously took a bit of reverse engineering, but basically I had to:

1. Find the path where Jose Chess and its embedded MySQL instance is storing the database. On Windows 7 it is - C:\Users\<UserName>\AppData\Local\VirtualStore\Program Files (x86)\jose\database\mysql\jose
2. Install MySQL and MySQL workbench to have a standalone instance of MySQL
3. Create a new database (schema) in MySQL workbench, and in that DB – tables to match the Jose Chess tables
4. Drop the Jose files into my wherever my standalone instance of MySQL is storing its files - C:\ProgramData\MySQL\MySQL Server 5.5\data\jose_huge_db
5. Restart standalone instance of MySQL
6. Run the query on the screenshot above!

Here are the files that need to be dropped around:

 

The only (but significant) disappointment is that Jose Chess is still storing game text (moves) in its internal binary format, so I can’t easily get to that data …

Creating Opening Repertoire in ChessBase

I already wrote about how to better manager your repertoire as well as why you need to build your repertoire in the first place, but this video (posted by “Robofriven” on ChessVideos.TV) illustrates nicely the details of doing it in a particular tool – ChessBase 9 in this case. The author also mentions Chess Position Trainer, so you definitely don’t have to be restricted to one tool; more important is that your repertoire is stored somewhere at all, and you have simple ways of it updating it.

 
http://www.chessvideos.tv/forum/viewtopic.php?t=4477

10 Reasons to Build an Opening Repertoire

I’ve been alluding to the importance of building an opening repertoire by talking about various methods of doing it, but I have not actually explicitly discussed why it is important to build up an opening repertoire, and put it into a database. Of course, you’d expect to get better positions out of the opening! But as a bonus, here are 10 other direct reasons why having a well defined repertoire is going to benefit your play and study:

1. You will save time on the clock during games, since you won’t have to think about which line to choose today
2. Every game played with a repertoire (online blitz, over the board, etc) – takes you closer to understanding your type of middlegame. Also, instead of playing random positions, you are now working towards a glorious goal of polishing your openings
3. You can ensure you study games of grandmasters that match your repertoire, rather than random ones
4. If you repertoire is in a database, you can give it to computer engine to blunder check
5. If you want to analyse a fun position – pick one from your repertoire, and you are working on an opening novelty!
6. You can test how well you remember your openings with a tool like Chess Position Trainer
7. If you have a chess study buddy, you can play practice games from certain starting positions, just like Botvinnik did with Ragozin!
8. If you have a coach, he can help to review your repertoire, rather than labour on building it for you from scratch
9. You have a better starting point when preparing for a game against a particular opponent
10. You can now focus on improving other parts of the game, such as middlegame, tactics, endgame, etc. Improving your repertoire will take care of itself, it is an ongoing process, but you have a baseline to work against!

You may have your own reasons, but the last one is a clear winner for me. Ultimately chess is a game for us to enjoy, and sometimes having little bit of discipline up front makes it more pleasant and fun in the long run, going back to my older post on how to enjoy studying openings.

Chess Assistant feature - Search for current pawn structure and material

Chess Assistant and Chessbase are both very powerful products, and it feels that the majority of chess players in the recent years have almost completely switched to Chessbase. I also have long ago decided that the convenience of the user interface even in various versions of Chessbase Light is more important than anything else in a chess package because it allows me to focus better on the 3 primary functions that I care most about:

1. Game entry
2. Viewing games
3. Engine analysis (and adding engine’s line to my analysis)

Yes, inability to easily resize the board has just turned me off from my good old copy of Chess Assistant 6, but every once in a while I still load it up, and I must say there are a few things that the Russian-based software company nailed back 10 years ago. For example:

1. Position Tree (integrated into the database)
2. Search

What is so fancy about “Search” in Chess Assistant? It is just more powerful! It

1. Remembers your previous search criteria (up to 10 or so)
2. Allows you to search multiple databases at once
3. Stores search results into datasets, and they are still around when you re-start the application
4. Is fast (again, goes back to positional tree integrated with the database, not stored in a separate lonely .ctg file Chessbase-style)

And here is the kicker, a very neat feature that allows you to better understand and study typical middlegames: “Search for Current pawn structure and material”:

That brings up a scary looking but otherwise very functional window like this:

It is very convenient to choose a database to search in, or multiple databases (something that is a traditional pain in Chessbase):

Searching hugebase then give a set of useful games with Black pawns on a5 and b5, and all 4 minor pieces still on the board:

It then remembers the dataset for future sessions

 

Sometimes it just feels that software makers don’t want to look at each other’s products! So 10 years later, some features are still better in one package, and resizing the board window is still problematic in its competitor. Or is it just that some things don’t fit into your original design, leaving you permanently screwed?

ChessBase in Silverlight

Chessbase finally has re-written their database software using one Microsoft’s latest technologies – Silverlight. You can “install” it by going to http://www.chesslive.de/. And it’s free to use! And it looks all pretty and shiny!

Ok, the first paragraph is full of overstatements:

1. this just gives the basics of functionality compared to their main products
2. It’s free to use if you run a non-free operating system, Windows. It seems to be using Silverlight 3, which is as far as I know has not been ported to Moonlight yet, so I suspect you won’t be able to use this interface on Linux today.
3. This just worked on my main PC, but on my laptop the install was a bit of a pain: it was first downloading “stuff” for a couple of minutes, and then informed me that I needed to upgrade Silverlight 2 to Silverlight 3. After I did that, my IE would begin to crash on startup, so I gave up on this whole Silverlight install thing in frustration. I suppose this complaint goes against Silverlight, not Chessbase.
4. “Paste position” did not work for me, I hope they fix it soon

But enough of whining, I think it’s great that more and more chess websites adopt Silverlight (The Week in Chess did that about a year ago). The Chessbase old Java-based online interface looked hideous and was barely functional. This one is pretty nice, and I can drag and drop UI panels around!

Improving Chess Analysis Skills – Bishop against Knight endgame

I am a believer in the approach of analyzing one’s games as a major tool for chess improvement, and already wrote a post about it. I keep a database of all games I played, and as I am bored – look back at my old endgames from many years ago. Most of them are more complicated than they appear on the first sight, but what I find even more interesting is that my old annotations from around 1999-2001 often contain more mistakes than the actual games. Mark Dvoretsky pointed out the phenomena that players are more relaxed during analysis than during the actual games, so annotations contain even the most obvious errors. He used Shirov’s games, but then also went on to give an example of how Kasparov missed mate in one while annotating Lasker-Steinitz game. I suspect in a lot of cases, including my own – the process of finding mistakes in old analysis has to do with

• computer engines having become stronger by the time you get around to double check your analysis
• having more time to focus on a position (Alexei Shirov is probably a bit more busy playing in tournaments than Mark Dvoretsky, I also don’t play as much now as I used to)

Medalen – Jiganchine, 2000

Black to move. Does 73… Bb8 win?

I played 73…g3!?, and after 74. Nd6 Bxd6 we both promoted our pawns, I got a drawn queen endgame with an extra pawn (which my opponent quickly lost). During and after the game in my analysis I was convinced that Bb8 wins on a spot. But actually – it leads to an even more forced draw. This must be some kind of optical illusion that makes you think that a bishop is doing a superior job to a knight in blocking a pawn, but they are actually equivalent as the next diagram shows.

74. Kb5! g3 75. Ne3 h5 76. Kxa5 h4 77. Kb6
Black to move. There is no win.
77… g2 78. Nxg2 Kxg2 79. a5 h3 80. a6 h2 81. a7 Bxa7+ 82. Kxa7 h1=Q 83. b8=Q = with a dead draw

Why you should not trust chess database statistics

Have a look at this position, which arises in Panov Attack in the Caro-Kann, or from some lines of Queen’s Gambit Declined.

1. e4 c6 2. d4 d5 3. exd5 cxd5 4. c4 Nf6 5. Nc3 e6 6. Nf3 Be7 7. cxd5 Nxd5 8.
Bd3 Nc6 9. O-O O-O 10. Re1 and now Black can play Qa5, which is a very rare sideline

Say, the database will tell you that 4 games have been played with this move, and black scored 50% with Qa5. The move itself sure does not look quite right, but Viktor Korchnoi himself played it, and the score seems not too bad. The 50% however is completely deceptive, have a look at another position from Brodsky – Maiorov, one of the games played in this line.

  White to move

White (a Grandmaster) is completely winning, but he played the completely inexplicable 47. Rd2?? dropping the rook on c4, and resigned a couple of moves later. So that 50% should have really been 75% in White’s favour! 4 games is really too small of a pool to rely on the numbers, so you should really look at objective value of Qa5 instead of relying on rather meaningless percentage from a database. I had failed to do that, so Black’s other loss is mine – from my 2002 game against Stephen Glinert! The title of the post is of course not fully true – sometimes you should check how well each side is scoring in a line before playing it, but it’s more important to understand the meaning of moves while building up your opening repertoire, especially if you rely on sidelines like Qa5.

10 tips for Analysing your Chess Games

One of the major ways of studying chess in our Russian “chess school” in the nineties was analyzing your own games. That got me into believing that this is the best way to improve your game. I started doing it without computer, on a piece of paper, but obviously over time tried to use databases as much as possible for maintaining and updating a collection of my games. I also helped several students of mine to learn how to analyze their games in effective ways, so here are a few suggestions that you might find useful:

1. Maintain a database of all your games. I keep several databases: my games with standard time controls, rapid time controls, internet games
2. As soon as possible after the game has finished – put down thoughts you had during the game. That will help you later to remember and understand the reasons for your mistakes.
3. Let the computer engine run through the game in blunder check mode – that way you’ll know immediately about the major blunders you and your opponent made
4. Identify the critical moments of the game. How many times does evaluation of position change, and advantage shifts from one side to the other?
5. Analyse the opening, update your opening repertoire if necessary. Evaluate the position after the opening, to decide whether your openings need “repair”
6. Do not just analyse in terms of variations. Give verbal evaluations of critical positions. If white is better – say why. That helps you to better understand the true meaning of each position. That also makes you stop looking at the computer evaluation and think on your own for a few seconds.
7. When you’re done analyzing – summarize your the game. Why did the game end the way it did? Where was it decided – opening, endgame, tactical blunder?
8. Over time – look at the trends in your games. Do you lose more points in openings or in endgames? Is there anything you can study in particular to improve those trends?
9. Go back to your games, even years after they were played. I do that just to practice my analytical skills, and very often I find surprising how much new little details I can discover (e.g. the endgame I thought was drawn – is actually winning, etc).
10. As already mentioned – don’t fully rely on the computer engine. Try to find moves and ideas on your own, and only then let the engine give you hints. It is ok to guide the engine, but make sure you’re still the driver.

Chess Position Trainer – test your opening knowledge!

I have written a fair bit about how you’d want to be careful about choosing the openings that you play and opening preparation in general. But how do you actually remember the multitude of lines that constitute your repertoire? If you carefully analyzed certain openings, and your repertoire consists of your original ideas – memorization likely won't be an issue. But if you never played a line before, and barely can think of a couple of plans – as soon as a position on move 10 comes up on the board, you'll realize pretty soon that:

• The devil is in the details – understanding positional ideas is great, but you still need to play specific moves to implement those ideas
• Move order matters
• some positions look similar, but the move you remembered and thought should be played in this particular position is actually good in a different sub-variation, but is inappropriate in this particular case

To test and refine my understanding of an opening – I usually play blitz games on ICC, look at them later and see whether I deviated from my own repertoire, or whether there is actually a gap in my repertoire that this game revealed. While practice games are super-useful, that type of training has some limitations:

• you have no control over what opponents will throw at you (you really did not want to test your knowledge of 1…b6 !)
• in the rush of a blitz game - you may not focus too much on playing the move that you had prepared (I already spent 30 seconds on the opening, I’d better move something quickly)
• once the game has ended - you also need to have the stamina to go back and compare your moves against what you had planned in the preparation

So there is a need for a special tool to test your opening knowledge! I have always admired the developer who put up a ton of effort into creating a free tool that lets you manage your opening repertoire and then practice it in a training mode – Chess Position Trainer. It always felt like a tool I would want to write myself if I had the skills and time. As it often happens with free software - the author had to fill the gap that no other free or commercial application would cover sufficiently for him.

However I have not used it for a while as it seemed that all my data was already in pgn and chessbase format, and my opening repertoire existed outside of Chess Position Trainer ecosystem. What was even worse – my repertoire had too many gaps (it still does), so I spent the last year going through books, databases and my old games, deciding what I should play against such and such line. Today I imported the opening repertoire I had created with Chessbase Light into Chess position Trainer, and here we go – I can practice my openings again!

In the training mode – the UI looks like this:

Training mode

A few points I like about its training mode:

• games are thrown at you relatively randomly
• Your progress is tracked (I have not dug into the details)
• The board is large and I like its simple look – often underestimated but very important, but some $100 packages got that part wrong.
• You’re told immediately whether you made a correct move (“correct” means – the move you yourself defined to be in your repertoire)

Sometimes it keeps throwing the same line at you until you actually know enter all the correct moves:

Training Mode is a pretty fundamental part of Chess Position Trainer – you can tell that from looking at training options (that’s a lot of Tabs!):

I would agree with what Stefan (the developer) wrote in this blog post – "this time I have already at hand a chess tool to study my material."

Outside of the Opening Training mode – the UI looks like this. This is one of the most well designed user interfaces for a chess program I’ve seen so far. It sounds silly, but buttons are large and convenient to press. It feels that the author actually uses this program himself, so he tried to make things easy for himself – and therefore, for any other chess player using his product. At the core of the software - the focus is as much on positions as it is on the moves, which makes sense for opening preparation.

Chess Statistics – results as White vs. Black

As I am addicted to statistics, I decided to count my losses in tournament games, and just as I expected – I lost a lot more games as Black than as White. Since the year 2000 – 31 losses with the Black pieces, compared to 13 games I lost as White. These kind of stats indicate that the opening, and the direction it gives to the game have a significant impact on your tournament results (surprise …). With better opening preparation, you can hope for better results, but this also indicates that having a solid repertoire as Black is generally more important, or else you will be losing a lot of games without any fight.

Following a similar pattern, my results as White are not that different – White wins more games. With ChessBase Light it is easy to generate such statistics, just pick all your games (or maybe first filter it by year to exclude your kindergarten games), and hit the `S` key.

Total breakdown as White:

And as Black:

This also shows that I played more games as Black (looking at the results – tournament pairings probably did cost me a few rating points here and there…). Of course, you need to keep a database of your own games to be able to do this kind of analysis.

Opening preparation – how to make a tree

I am not a big fan of opening preparation, it’s never been very easy for me. I enjoy studying the endgame a lot more. At the same time, I must admit that without proper opening knowledge – every game can potentially turn into a lot of suffering. In some openings you may get away with just knowing general ideas, but generally you can’t get away without knowing some precise variations.
Around the year 1998, I lost a speed chess game to a master. I lost it in an embarrassing manner, by getting my rook trapped on move 10 in the opening. I forgot the move order in exchange variation of Slav defence, and there was no return. After the game, the opponent looked at me with understanding, and gave me a very valuable advice (ok, one of the many he gave me). He said – “For every opening you play – you must have a tree. Does not matter how you do it, on paper, on computer, you must have it recorded somewhere”. Indeed, if you don’t bother to record it once, how likely are you ever going to REMEMBER thing?
Computers make it really easy to do this – you can create a database of games that looks like this (you can create it in Chessbase Light 6, and then load in Chessbase light 2007 for viewing – both are free).

In the body of the game, you enter the variations, and then if you do File-Print-Print Repertoire, you can get a nice printout like the screenshot below. It looks just like your own NCO! Store it under your pillow and that way you won’t forget your moves after 23…Rc4 in the Panov attack…

I recently wrote another post on opening preparation that has more to do with the psychological aspects of studying the opening.

Chess on linux

Chess on Linux, well, is possible. Since my windows installation went down last week, I've been experimenting with running different chess (and other) software on Ubuntu 7.10 and learning whether migrating to Linux is really as painful as people say. A few discoveries:

- Wine does miracles. Both Chessbase Light 6.0 and Chessbase Light 2007 run in Wine (the 2007 version must be installed via Wine, the ancient 6.0 version can run from a windows partition)

- Blitzin does work from Wine, but moves get input with a bit of delay (very fast UI is the main benefit of Blitzin)

- Jin is nice, but I lost a 3 minute game on time, and I want to blame it on the 0.5 delay that the UI has for displaying the moves - you hear the sound of opponent's move first, and only later see the actual move on the board. Note that I had tried the web based version of Jin before on Windows, and it worked perfectly.

- There are a lot of tools that are still missing and that I have not tried to find an alternative for yet, like Palview, Windows media recorder, and Windows live writer - I pretty much relied fully on them for making blog entries and videos for my youtube channel.

All in all, the amount of free software on Ubuntu, and decent interoperability with windows programs really put me on the fence as to whether or not to go back to windows full time for my home computing needs.

Chess Opening Explorer (web 2.0)

In my previous post I described the issues with different file formats for storing chess games for viewing on a local PC. The need to constantly refresh references databases by downloading weekly game updates, copying and pasting annotations between databases to keep notes up to date, re-building opening trees, etc only add more to the challenges of organizing one's game collections. Maybe web 2.0 is the answer to all those issues? chessgames.com has become sort a youtube of chess games. Their latest feature - opening explorer is also quite neat! Some lines require paid subscription to be able to see beyond move 3, but this line of Sveshnikov that I am interested in is available up to move move 15. Quite a teaser; just enough to get me interested in their services.

Another interesting feature is collections of games - sort of like playlists in youtube - lists of games selected by users. For example - here is a collection of games from one of Kasparov's volumes. Paid members can download entire collections in pgn. I like where this is going! Having discovered RSS only last year, I can't help thinking that next step would be for websites like TWIC to provide game updates in rss feeds. That way anyone who wants to get all games played in the last few months can conveniently get their updates via an RSS client, instead of having to manually click and download on a hundred zip files on their page.