Leaf through any of W.G. Sebald’s core works of fiction – Vertigo, The Emigrants, The Rings of Saturn and Austerlitz – and you will find indistinct and frequently indecipherable photographs inserted into the text. None bear captions although some are numbered, apparently at random. Their contents are similarly mysterious, often even when taken in context. In other words it is as difficult to conjecture their relationship to the writing as it is to conjure up what they actually describe. What does an image of a window showing only sky have in common with an image of a diary dated 1913, or of a tumbledown tower at a popular Victorian seaside resort? How does a picture postcard of a dyke beside a ruined windmill, a reproduction of a medieval woodcut depicting souls in torment, or a portrait of a young boy with a spade in a manipulated mock-up of Pisa’s Campo Santo, connect with the text, or with one another?

Historically, photography has had two fundamental characteristics. One is its link with reality and the other with memory. In this sense photographers are always witnesses and their photographs are their testament to the fact that ‘I was here, in this place, with that person you see, at a particular moment in time.’ Yet Sebald, who wrote in all his works about history and memory, used photos for the opposite purpose, to fracture and fragment those links. In the words of his long-standing friend and colleague Clive Scott: ‘Sebald’s photos don’t fill holes, they create them, and his histories (of herrings, or Chinese emperors, or Edward Fitzgerald) make us feel less sure of ourselves.’ As do, of course, his books. For, whether ostensibly about reality (through travelogue, in The Rings of Saturn, Vertigo), biography (The Emigrants, Austerlitz) or history (On the Natural History of Destruction), along with many literary essays and academic papers, Sebald’s technique was to subvert not only established genres but also the possibility of conclusively dividing one from the next. He was an author of fiction that read like fact – and, sometimes, of history that read like fiction. He included photographs as a further form of allusion that, like his writing, evoke recollections in the reader, true or not. And whether or not as records of actual events, or as integral to the stories we all tell ourselves.

‘I am a history, a memory inventing itself.’ Not a phrase of Sebald’s, in fact, but of the Mexican writer Octavio Paz. And we have all learnt, if not to doubt, at least to question our memories. Is this a memory or a recollection of the act of remembering? Is it my personal memory or that of something someone once told me,or I read in a book or saw in a film? How is it that memories can have the qualities of dreams, with the preposterous appearing unexpectedly but unsurprisingly in the midst of everyday remembrance?

A snapshot: my first memory of seeing Max Sebald is banal enough – at a party held in his honour at the home of his then publisher. The drawing room is on the first floor and I am standing near the doorway. A parting of the waters as Susan Sontag enters, dressed entirely in black, her characteristic white bang amid the long black hair like the crest of an exotic bird. She pauses beside me, and I wonder if she has picked up something of our conversation about Central America that interests her. Not at all: casing the crowded room, she immediately asks me to direct her to Max. Having only just arrived myself, I glance randomly around until I see the innocuous pink-cheeked, white-haired gentleman seated in the corner of a sofa in a corner of the room, avoiding social conversation by peering intently at an adjacent bookshelf, and I direct her to him. Sontag crosses the room and literally sits down at his feet.

Sebald deploys photographs to continually subvert his readers’ expectations. Carefully selected and laid out (by himself: this was too important a task to leave to the designer or publisher) Sebald seeks, often playfully, to insert the exotic into the everyday. In Vertigo he describes a climb up Burg Greifenstein, overlooking the Danube, where, he writes, ‘I had first visited the castle in the late 1960s.’ Accompanying the account is the image of a giant cactus sprouting over the walls of a whitewashed villa typical of Latin America. Does any dwelling less resemble an Austrian castle than a Mexican hacienda? Why is the description of the one accompanied by an image of the latter? The photograph, it would seem, has nothing to say regarding the story; the effort becomes the reader’s to discover what it ‘really’ has to tell us. Its irrelevance to the ‘reality’ of the story is only the start.

The way to the castle/hacienda has already been paved by the immediately preceding story of Sebald’s visit to a long-term inmate of a mental asylum, Ernst Herbeck: ‘He was wearing a glencheck suit with a hiking badge on the lapel,’ Sebald writes. ‘On his head he wore a narrow-brimmed hat, a kind of trilby, which he later took off and carried beside him, just as my grandfather often used to do on summer walks.’ The grandfather roots the asylum visit in reality: Sebald preferred to spend his childhood free time with his maternal grandfather, his chosen father substitute. The grandfather is the most ‘real’ – in terms of being the most present – of Sebald’s family members in his books, perhaps also in his memory. Despite the detailed physical description, the portrait shows the torso of a suited man, his head – hatted or not – cropped from the frame.

In looking for a head in a trilby – or a trilby in the hand of a suitably dressed hiker – we find instead a hand nervously clenched to a man’s side, clothed in an unchecked three-piece suit and tie. The great Hungarian photographer Robert Capa defined a good photograph as one that leaves the viewer yearning to learn what is taking place just outside the frame. Here the message is that the camera doesn’t lie; it simply doesn’t reveal.

Perhaps the association with the grandfather at least provides a clue. According to the obituary in the Guardian, as a young boy Sebald came across his father’s photo-album at home. In it ‘… he found pictures taken during the Polish campaign in 1939, and he sensed that something in the grinning German soldiers and boy scout atmosphere of the campaign, ending with the torching of villages not unlike his own Bavarian home in Wertach im Allgäu, hinted at the meaning of the destroyed buildings, silences and absence of memory around him’. His father, a career soldier who enthusiastically rallied to the Nazi cause, did not discuss the war with his family. As an adult Sebald was to research, discover and infill some of the missing memories through a laborious process of reconstruction and rewriting.

It was the first photo-album Sebald had ever encountered, and he began a postcard collection. When he moved to England in the 1960s, he became an avid collector of found photos and a deliberate amateur in taking his own pictures: by eschewing the well-known artist and their iconic images, he was placing himself firmly on the side of the snapshot, the caught moment and the family album. The use Sebald made of them is more ambiguous but deeply deliberate.

Another snapshot: the first time I met Max was at a summer school at the University of East Anglia. He had been in earnest and dynamic conversation with the translator Anthea Bell about her ongoing translation of Sebald’s On the Natural History of Destruction. It was extremely clear he took a fastidious interest in the possible variants almost on a word-by-word basis, and that every association merited serious consideration. Anthea welcomed and engaged with each suggestion without getting into an argument. Her consummate diplomacy rendered it a discussion of equals; more, even, a meeting of minds. Over lunch I asked Max why he persisted in writing in German, a language he otherwise had had no cause to use in his working life for some 35 years. With humour, he acknowledged that his German must indeed have become a little arcane but that it was his German for all that. He had a sense that the language needed cleansing of some of its former associations and reclaiming as that of the great literary figures who had peopled his earlier studies and publications. Yet his archaic style has repeatedly been advanced as the reason why his books sell less in German-speaking countries than in England: ‘Nobody writes like that in Germany any more nowadays.’

As if by contrast, there was nothing precious, or even professional, about Sebald’s own photography. He used the cheapest colour film, shooting book plates alongside holiday snapshots, to be converted into black-and-white prints by the photographer Michael Brandon-Jones. Max instructed him in how to crop or manipulate them, or took a pair of nail scissors to them himself, famously removing a row of parked cars between the bed-and-breakfast hostels and the sea at Lowestoft. Sebald was meticulous in his intentions, arriving at the UEA darkroom with a text neatly typed on A4, and with spaces carefully inserted for a vertical or horizontal, square or strip image, all done to scale for book-sized reproduction. Brandon-Jones could not understand Sebald’s German original so at the time could not tell whether the visual matched the verbal narrative. He gave his opinion on what the photos were doing: ‘They were there both as clues and also to confuse the reader. Max was the first writer, I believe, ever to use photos to deceive in that way.’

Already in The Emigrants, his earliest work of fiction, Sebald is cutting and pasting photos, some of which we assume to be portraits – naturally enough of the four exiles who are the subjects of the book. Sebald wrote to his first translator, Michael Hulse, saying: ‘I can assure you that all four stories are, almost entirely, grounded in fact …’ but there is little in the photo-images to endorse this. There is an end-of-civilization aspect to the postcard of the idyllically named Hotel Eden at Montreux, as there is to the aerial view of a bombed British city, Manchester. Here, however, each image relates, if not to the lives of actual emigrants, at least to actual post-war places.

Two years later in The Rings of Saturn – Sebald’s pseudo-realistic account of his long walk around the East Angian coast – photographs are deployed differently. Not a train in the chapter on the Princess of China’s steam locomotive, for example, any more than a merman in that on the Orford Ness dungeon where the poor creature was incarcerated. Instead we have wintry expanses of marshland and estuary, a stretch of shoreline which may or may not be at the place under discussion (Dunwich), plus a picture of some hatted gentlemen who may – or may not – have been standing around Eccles Church Tower (they weren’t) as it crumbled to sand and was consumed by the invading tides. Whatever these oddly cropped, poorly reproduced and fogged reproductions are doing, they are not visually documenting the text.

Another snapshot: the year after W.G. Sebald’s death, I become director of the British Centre for Literary Translation, founded by him in 1989. Before I can move my books on to the racks in his former room, a colleague knocks on the door and introduces herself. She is looking over my shoulder as she speaks and finally spots something of interest. ‘Ah,’ she says. ‘The sherry glasses. I’ll have those.’ So saying, she removes a box from the top shelf. As she departs she announces, ‘They belonged to Max so I am taking them with me.’ The homage continues.

Sebald believed the legacy of the great nineteenth-century novel to be dead. Following a century of horrific violence, literature was bound to reflect fragmentation, and photographs could hint at a synthesis between truths and fictions. Rather than the set-pieces of epic Napoleonic battles reprised at the start of Vertigo, the scrapbook accumulation of images in Sebald’s work offers us something more approaching a Kurt Schwitters collage. There is no history that does not incorporate (auto)biography, and photography can no longer be used as reliable evidence. Without history, or a common collective memory, then indeed we are left with Octavio Paz’s statement of the individual: I am a history, a memory inventing itself. Max would have been amused at the story of the glasses. After all, as I was shortly informed: ‘They were left not by the first but the last director of the BCLT, years after Max.’ One person’s memory becomes another’s invention and we all believe what we desire.

Originally commissioned in 2011 for Radio 3’s The Essay, aired each week night at 22.45. Audio available at: www.bbc.co.uk