I worked on a project control system in 1999. It came from a software house and used C programs and Oracle Forms running against an Oracle 7 database.
The system was not Y2K compliant so it had to be modified. It was moved to Oracle 8 at the same time.
New versions of the C programs were sent to us and I compiled them on a UNIX test server. I also upgraded the database using export and import.
Testers were assigned to check that the updated software worked properly. They reported that one of the C programs did a core dump when they ran it. I passed this fault on to the software house.
They said that this program ran on other customer sites without falling over and suggested that there must be something wrong with my installation. I replied that I had followed their instructions when I installed the software so there was nothing wrong with my installation either. This went on for a couple of weeks but both sides refused to back down. One option suggested was that the software house should come and investigate the problem on-site at our expense. If they had done this and found a fault in my installation, it would have been embarrassing so I decided to get to the bottom of the problem myself. This is not always possible but fortunately, on this occasion, I had already done some C programming.
One evening when my colleagues had left, I took the source of the C program, added a few debug statements, recompiled it and ran it several times. Three hours later I found the cause of the problem. At the start of the program, a file was opened. When this happens in C, that file is subsequently accessed via a pointer. A bit further on, the program copied a 10 character data item into a 9 character data item. Our compiler had placed the file pointer next to that 9 character data item. So when the 9 character data item was overwritten, the spare character corrupted the file pointer. Several lines later, the program tried to use the corrupted pointer to write to the file and promptly did a core dump. The program only worked on other customer sites because they used different compilers. These different compilers arranged items in memory in such a way that no damage was done by the superfluous 10th character.
I reported my findings to the software house. They supplied a new version of the program which changed the 9 character data item to a 10 character data item. I recompiled it and it ran successfully.
However, even though I had helped the software house to fix a bug in one of their programs, they still weren't happy and complained to my manager that I should not have been looking at their source code. There's no pleasing some people, as the saying goes!