The Battle of Midway Roundtable

 

http://www.midway42.org/

 

 

The victory at Midway was won due to a number of essential factors, the first of which was the remarkable success of the codebreakers.  This brief article explains how that success came about.

 

Revised 17 August 2005

 

 

"THE SILENT WAR AGAINST THE JAPANESE NAVY"

 

by Captain Duane L. Whitlock, U.S. Navy, Retired

Copyright 1995 by Duane L.  Whitlock, Naval War College Review, Autumn 1995, Vol. XLVIll, No. 4

 

- - - - - - -

 

Forward:

 

            Captain Whitlock entered the service in June 1935, became a radioman, and was specially trained to intercept Japanese naval radio communications.  He served as an intercept operator in Hawaii and on Guam, became a self-trained analyst in 1940 while stationed in the Philippines, and began producing intelligence estimates on the Japanese navy's air, surface, and subsurface operations.  Evacuated from Corregidor in March 1942, he continued to produce intelligence on the Japanese navy throughout the war, first from Australia and then from Washington, D.C.  Commissioned temporarily in 1943, he received a permanent commission after the war, attended George Washington University, and served the rest of his career in the naval cryptologic and intelligence communities.  After attending the Naval War College in 1958, he served as a cryptologic advisor on the staff of the Commander in Chief, Pacific Fleet.  Retired since 1967, he is the author of Critical Thoughts and Notions (Danville, Calif: Omega, 1988).

 

This article is adapted from a paper delivered on 11 August 1994 to a conference on "World War II in the Pacific" at Crystal City, Virginia, cosponsored by the Naval Historical Center, U.S. Naval Institute, Naval Order of the United States, Naval Historical Foundation, Marine Corps Association, American Society of Naval Engineers, and Marine Corps Historical Foundation. 

 

- - - - - - -

 

            On the day the bombs fell on Pearl Harbor, the U.S. Navy, or at least a tiny segment of it, had had the Imperial Japanese Navy under attack for about twenty years.  The attack was, of course, a silent one, of which the Japanese were totally unaware.  It began in 1921, when the Office of Naval Intelligence (ONI) surreptitiously acquired a photographic copy of the "Imperial Japanese Navy Secret Operating Code - 1918." [1]

 

            The code was in essence a dictionary containing a hundred thousand entries, and it took five years to translate; only two Japanese linguists were available, and there was no particular urgency or incentive attaching to the project.  After all, having a code book is of no great advantage if one does not have access to messages being encrypted in that code.  ONI at the time did not have that access, and gaining it was not a simple matter, because the Japanese use a different telegraphic code for radio communications than does the rest of the world.  Keyed to the Japanese alphabet, or syllabary, known as Kata Kana, it contains nearly twice as many dot-and-dash combinations as the Morse code: Kata Kana, sometimes referred to as "hen tracks," is a simple pictorial means of phoneticizing the Japanese spoken language.  In 1923, the Chief of Naval Operations (CNO), perhaps unaware of the nature of the Japanese telegraphic code, requested that Asiatic and Pacific fleet radio operators listen in their spare time for enciphered foreign radio messages.  [2]

 

            To what extent this invitation served its purpose is unknown, but several Navy and Marine Corps operators in the Far East did teach themselves to recognize and intercept Japanese radio communications.  One of these operators, Chief Radioman Harry Kidder, was serving in the Philippines.  With the help of the Japanese wife of a shipmate, he learned the Kata Kana syllabary, taught himself the telegraphic equivalents of all the Kata Kana characters, and began to intercept Japanese messages. [3]  Whether anyone in Washington was aware of his accomplishment at the time is not clear; that it paid enormous dividends in years to follow is an indisputable matter of record. 

 

            A few other operators on the Asiatic Station somehow learned to write Kata Kana characters and copy Japanese messages with a pencil, though none ever gained the stature of Harry Kidder.  Initially, intercept operations were uncoordinated and piecemeal, carried out on board only the one or two ships of the Asiatic Fleet that happened to boast a self-trained Kana operator.  In 1924, to obtain more legible copy, and eliminate writer's cramp, Washington purchased a few specially designed Japanese typewriters. [4]

 

            In that same year, the first shore-based intercept station was established, in the American consulate in Shanghai. [5]  Its primary target was the diplomatic radio network serving the numerous Japanese consulates throughout China.  In 1927, this network became the prime target also of self-trained Marine Corps operators at the Marine detachment in Peiping (as it was then known, modern Beijing).  Originally designated, in the phonetic alphabet of the time, Station ABLE, their unit was disestablished eight years later when both the Japanese threat to the city became too pressing and personnel limitations caused the Marine Corps to withdraw completely from intercept work.  Responsibility for the diplomatic network was shifted to the headquarters of the Fourth Marine Regiment in Shanghai, which possessed an enclave of career Navy intercept operators trained in Japanese traffic.  Their unit became the new Station ABLE. [6]

 

            In 1926, ONI finally finished translating the 1918 Japanese Navy code book acquired in 1921.  Code book updates were obtained, again surreptitiously, in 1926 and in 1927.  Given the cover-name "Red Book," the translation was turned over to the Director of Naval Communications (DNC), for whom I served, as a constant incentive to build up a radio intelligence organization. [7]  Having the code but not the skilled personnel needed to intercept Imperial Japanese Navy (IJN) messages in significant quantity, the DNC, with the assistance of ONI, set about carefully screening and selecting a few well qualified fleet radio operators.  To train them in Kata Kana and the Japanese telegraphic code, a school was set up in a specially constructed blockhouse on the roof of the old Navy Department building in Washington, D.C.  (Graduates of this school were eventually venerated as the "On the Roof Gang.") The search for a qualified instructor turned up Chief Kidder, who happened to be serving in the Navy Department communications center.  The school opened in October 1928, and in the next eighteen months Kidder trained and graduated three classes. [8]

 

            Eight graduates were sent to Guam, where, under the tutelage of another self-taught Kana operator, Chief Radioman Malcolm Lyon, they established the Navy's second intercept site, Station BAKER. 

 

            Scarcely had this site come into existence when, in 1930, the nine enlisted men stationed there scored an intelligence coup of historic significance.  Monitoring Japanese naval communications, they were able to deduce from patterns alone that the entire fleet had sortied from Japan and was engaged in an exercise of massive proportions.  The Japanese navy had even managed to call up all its reserve ships and personnel and send them to sea with the rest of the fleet, all with such secrecy that the U.S. naval attaché in Tokyo was unaware of anything unusual. [9]  The material intercepted by Station BAKER slowly made its way to Washington.  There the small cryptanalytic staff labored on it for many months, using the Red Book, ultimately decrypting enough to discover that the activity detected by Guam had been a triennial "Grand Maneuver," in this case a test of a Japanese navy plan to support an invasion of Manchuria—as would take place the following year. [10]

 

            This episode pointed up the dire inadequacy of the funds and facilities devoted to the intercept and processing of Japanese navy radio communications.  To serve the two senior admirals in the Pacific theater, major decrypting units were needed in Hawaii and in the Far East, with many trained intercept operators, cryptanalysts, and Japanese linguists.  Developing a radio intelligence organization of this scope turned out to be a long-range project, and it was not completed before the outbreak of the Pacific war.  In fact, were it not for the nudge given in 1930 by Station BAKER, there is substantial reason to believe it would not have been ready in time to contribute to the battles of Coral Sea and Midway.  As it was, the Guam operators had alerted authorities to the possibility of deriving intelligence from enemy communications without actually decrypting the messages. 

 

            Notwithstanding, a firm decision was made to explore that potential on the occasion of the next Japanese navy Grand Maneuver, which was expected to occur in 1933.  This decision was given considerable impetus in December 1930 when the Japanese navy superseded its 1918 code book, thereby eliminating the American ability to decrypt its messages.  The Red Book was now useless for all but historical purposes, and ONI found itself unable to obtain the new code in the way it had the old one.  Accordingly, the new book had to be reduced by cryptanalysis, a process that consumed most of the next five years. [11]  Admiral Upham, the Commander in Chief of the Asiatic Fleet, therefore assigned his staff radio intelligence officer, Lieutenant Joseph Wenger, to derive all possible intelligence on the expected exercise by means other than cryptanalysis.  Wenger arranged to receive the intercepts made by stations ABLE and BAKER and also some temporary sites. 

 

            The IJN maneuver occurred as predicted, and Lieutenant Wenger spent the next six months performing after-the-fact traffic analysis.  That effort produced a 115-page report laying out in considerable detail the composition and disposition of the forces involved, including the identity of the individual ships and commands.  His report was sent back to Washington, where three years later, enough of the new code (the "Blue Book") had been recovered to affirm that Lieutenant Wenger had been essentially accurate. [12]

 

            Admiral Upham did not wait for that assessment.  He was so impressed with Wenger's work that he immediately made a strong representation for a radio intelligence center in his command.  Specifically, he requested that an intercept unit complete with a decryption center be located in what war plans referred to as the Ultimate Defense Area of Manila Bay, with the mission of preventing surprise attack. [13]  This recommendation is historically significant for two reasons:  it was undoubtedly the first time that a U.S. Navy fleet commander ever requested that a decryption center be placed under his command; second, it marked the beginning of a willingness in the U.S. Navy to amend the long- standing precept that the military commander should always be guided by estimates of enemy capabilities, never of intent. 

 

            Implementation soon began.  Three graduates of Chief Kidder's first class and two from his second had been sent to the Philippines, where late in 1929 they attempted to set up an intercept station at Olongapo.  Unfortunately, they were delayed by having to assume primary responsibility for all regular Navy communications in and out of the base there. [14]  As a result, they did not really get on with intercept duties until August 1932, as Station CAST.  Station CAST was augmented as recommended by Admiral Upham; it was transferred from Olongapo to Mariveles and then to the Navy Yard in Cavite.  In mid-October 1940 it would finally establish itself in a special tunnel built for the Navy at Monkey Point on Corregidor, two months later absorbing the mission and the personnel of Station ABLE in Shanghai, which was closed.  [15]

 

            In the meantime, in 1935, a new intercept site, Station HYPO, was opened on the windward side of Oahu, and a decryption unit was established in the basement of the administration building at Pearl Harbor.  In June 1939, however, the Japanese navy once again changed its operating code, and the nine years of cryptanalytic effort that had gone into recovering and exploiting the Blue Book went down the drain. 

 

            The new code, designated JN-25 by the U.S. Navy, was of a drastically different type than its predecessors.  Made up of five-digit groups, it allowed for forty-five thousand entries in each of two dictionaries—one set up alphabetically for encoding, the other arranged numerically for decoding.  For the cryptanalyst, who had to gain entry to the code through the numerical dictionary, this provision had the effect of scrambling the alphabetical order of the code, thereby complicating the process of assigning meanings to individual code groups.  But the Japanese had erected an even greater barrier to that entry by enciphering each already encoded message by adding arithmetically a different and arbitrary five-digit number to each of the five-digit code groups contained in the message.  Generated randomly, these arbitrary enciphering groups were drawn systematically, from a separate book and were added, one for one, to each of the basic code groups awaiting encipherment.  "False addition" was used to combine these arbitrary groups, or "additives," to the basic code groups.  In false addition, no numbers are carried over; if the additive 49238 were added to the code group 77739, for example, the result would be 16967, which would be the five-digit group that was actually transmitted.  Inasmuch as there were eighteen thousand of these additives in the enciphering book, there existed a potential to encipher any one of the forty-five thousand basic code groups in eighteen thousand ways. 

 

            The system seemed unbreakable: one needed the additives in order to recover the basic code groups but had to have the basic code groups to find the additives.  The cryptanalyst's only hope was to sift all intercepted messages very carefully, hoping that Japanese code clerks would eventually make enough careless mistakes to permit entry into their system. 

 

            Responsibility for breaking JN-25 was shared between the Navy Department (now Station NEGAT) and the decryption unit at Station CAST, on Corregidor.  The unit in Pearl Harbor, Station HYPO, was assigned to work on the Imperial Japanese Navy's high level cipher used between admirals.  That code was never broken, mainly because the infrequency of its use denied cryptanalysts a sufficient depth of message traffic. [16]  Depth was not a problem with respect to JN-25, but, even with the opportunities provided by one or two errant Japanese code clerks, the very nature of that code made progress extremely slow and tedious.  In a division of effort, Corregidor was to attack the code on a current operational basis; Washington was to follow up several weeks or months later, with the primary aim of building up the dictionaries to support operational exploitation.  Between June 1939 and December 1941, Washington did decrypt a few JN-25 messages, but they provided little insight into the current intelligence picture.  The Japanese also made several changes in the code during that period, which put Corregidor and Washington out of sync. 

 

            Three days after the Pearl Harbor attack, the decryption unit of Station HYPO joined in working on JN-25, but it was not until 16 March 1942 that the combined recovery effort finally allowed Corregidor to issue its first complete decrypt of a current Japanese message.  That message revealed that the Japanese had assigned the geographical designator "AF" to the island of Midway. 

 

            Until that date and since June 1939, the only current radio intelligence available on the Japanese navy had been produced by methods short of decryption—that is, by traffic analysis. [17]  Most of that intelligence was being produced by a few self-taught enlisted analysts who had moved up from intercept duties.  Much of it was of a technical nature, relating, to the identification of encrypted call signs, which the Japanese changed periodically, and to the recovery of the enemy's communication plan.  In pursuing these tasks, the analysts learned considerable amount about how the IJN was organized, and they periodically produced and updated tables of organization (i.e., orders of battle) for its entire fleet. 

 

            For years, information of this nature had been included in the monthly reports that each intercept station was required to file.  At Station CAST, that practice began to change in the summer of 1941 when Lieutenant Jefferson Dennis, detached from duty as the Asiatic Fleet radio intelligence officer, arrived on Corregidor to await transportation back to the Navy Department.  A very capable analyst, while at Station CAST he began to produce a brief daily summary on the location and activity of various Japanese ships and aircraft as deduced from communications patterns; he sent it by radio to the Commander in Chief, Asiatic Fleet.  Studying his methods, the unit's three enlisted analysts began to produce a few items of current intelligence, which Dennis included in his summaries.  Initially, these items related only to events of the current date, correlated wherever possible with previous estimates, but soon a new dimension was added—interpreting enemy intent. 

 

            The change came about when one of the enlisted analysts noted an intriguing combination of addresses on a short encrypted message; it led him to suspect that a division of destroyers was departing from Taiwan to the island group of Palau, east of the Philippine island of Mindanao.  Lieutenant Dennis, after careful thought, tended to disagree, stating that Japanese ships destined for Palau normally proceeded by way of Saipan (due east of Manila) or Truk (over thousand miles east of Palau).  Two days later, however, a PBY reconnaissance aircraft flying out of Cavite reported three Japanese destroyers two hundred miles east of Manila on a southeasterly course toward Palau.  Thus it became apparent that traffic analysis had some potential to predict enemy intent, but because JN-25 was not yet yielding current intelligence, there existed no way to assess the limits of that potential. 

 

            When Lieutenant Dennis departed, the enlisted analysts of Station CAST continued to originate periodic intelligence estimates, dispatching them through the Commandant of the Sixteenth Naval District to the commanders in chief of the Asiatic and Pacific fleets, Station HYPO at Pearl Harbor, and CNO.  Following its lead, HYPO began filing similar estimates, through the Commandant of the Fourteenth Naval District.  In the fall of 1941, both stations detected and reported several organizational changes within the Japanese navy, one of which saw the consolidation of all land- based air units into a new air fleet consisting of several newly formed flotillas. 

 

            In October, the analysts on Corregidor incorporated all of these changes into a new order of battle, which was submitted to the Asiatic Fleet for transmittal to commander in chief of the Pacific Fleet and CNO.  In his endorsement, the Asiatic Fleet intelligence officer stated that the report could be construed in no way except that the Japanese navy had assumed a wartime disposition. 

 

            In early November 1941, those same analysts reported that Japanese merchant ships were being inducted into the navy in alarming numbers and that some 250 of those ships had been sent to ports bordering the Taiwan Strait, just north of the Philippines.  Then, in late November, traffic analysts both at Pearl Harbor and on Corregidor reported that two powerful Japanese task forces were approaching the Philippines, from the north and the east; on the basis of communications patterns, the report laid out the composition of both of those forces in considerable detail.  Corregidor, unfortunately, made the mistake of estimating that the large Japanese carriers were still in home waters, although none of them had been noted communicating for several days. [18]

 

            While this error was to contribute virtually nothing to the debacle at Pearl Harbor, it did teach both the producers and users of radio intelligence a valuable lesson about the limits of traffic analysis:  it has little if any effectiveness against a target that elects to observe radio silence. 

 

            Cryptanalysis, of course, is also hampered by that stratagem, but only to a lesser degree, for if cryptanalysts have broken the enemy's code, they can read messages being broadcast to the target "in the blind" (i.e., without established two-way contact).  They can do so provided, of course, that traffic analysts have broken the enemy's call-sign system and can identify the intended recipients.  In this regard, most decrypted messages are never fully intelligible until their encrypted call signs and addresses have been deciphered.  The complementary nature of these two basic forms of analysis became more and more apparent as the Pacific war progressed. 

 

            The first indication of Japanese intent with respect to Midway, for instance, appeared in a traffic analysis report filed by Corregidor in early March 1942; that estimate was backed up three days later by the JN-25 decrypt assigning the "AF" designator to the island.  By that time, traffic analysts on Corregidor and in Hawaii were reporting almost daily on the buildup of ships and aircraft at Truk and Rabaul.  These reports collectively revealed quite clearly a Japanese intention to launch a major thrust into the Coral Sea and probably toward Midway.  They also provided the basis for the estimate given to Admiral Chester W.  Nimitz by his intelligence officer, Edwin T.  Layton, that the Japanese had amassed in the Truk area some three hundred ships for impending operations.  (After the war, the Japanese indicated that the figure had been 280.) [19]

 

            On 26 May 1942, “it was estimated that 60 percent of Imperial Navy traffic was being intercepted (by the U.S. Navy) and 40 percent read, although the content recovered from the typical message averaged only about 10- 15 percent.” [20]  Obviously one does not get much intelligence out of a fifty-word message if only perhaps eight of the words can be read.  Although the ability to read Japanese naval messages was expanding rapidly just before the battle of Midway, that less than fifty of the 280 ships later acknowledged to have been present were ever mentioned in JN-25 decrypts points to a subtle limitation as to what intelligence can be derived through cryptanalysis. 

 

            Throughout the entire war, the Japanese navy never attempted to disseminate a complete operation order by radio; apparently, as in the U.S. Navy, such orders were always delivered by hand.  While JN-25 partial decrypts relating to the battles of the Coral Sea and Midway made tantalizing reference to operation orders and to such task organizations as the "striking force," "assault force," "Moresby force," etc., the composition of these groups had to be deduced almost exclusively from traffic analysis. 

 

            In fact, at the outset of the war, traffic analysis was, as it had been for many years, the only source of current intelligence bearing upon the strategic posture and the disposition of the surface, subsurface, and air elements of the Japanese navy.  As cryptanalysis began to catch up with current events in 1942, it started to add to the traffic analysis picture the timely and precise details essential to achieving tactical advantage.  The shooting down of Admiral Isoroku Yamamoto on 18 April 1943 is a classic example. [21]  However, as has not been well understood by historians who have highlighted the many tactical successes, cryptanalysis made a rather limited contribution to "the big picture" in the "silent war" against the Japanese navy.  Had it not been for the considerable number of victories mutually achieved by these two analytical methods in the silent war, the shooting war in the Pacific would have taken a far different and much more painful course. 

 

            A vital point that should not be overlooked by historians and students of the war with Japan is the fact that something more than twenty years was required to bring on-line the radio intelligence organization that ultimately gave commanders what was perhaps the greatest strategic and tactical advantage in the history of naval warfare.  That advantage shortened the war immeasurably, saved many lives, and reduced substantially the material and financial burden imposed upon American taxpayers.  That such an organization ever came into existence stands as a monument to the foresight of two or three flag officers, not the least of them Admiral Upham, as previously noted, and also, at a much earlier date, Commander Battleship Force, U.S.  Pacific Fleet, who in a letter of 16 May 1921 to CNO recommended that "all rated radiomen be given sufficient instruction in receiving Japanese Language Code to at least be able to recognize it when copied." [22]

 

            The latter recommendation is likely to have come to naught were it not for the singular initiative, imagination, persistence, and ingenuity applied to it by Lieutenant Laurance F. Safford.  One of the first U.S. naval officers to specialize in the new field of cryptology, he was largely responsible for introducing the art of cryptanalysis into the Navy.  Heading the newly established Cryptographic Research Desk in the CNO staff (or OPNAV) Code and Signal Section in 1924-1925, he spearheaded the Navy's attack against the Japanese diplomatic codes, provided rudimentary training in cryptanalysis to one or two specially selected junior officers, encouraged the ad hoc Marine Corps and Navy intercept efforts taking place in the Asiatic Fleet, and took steps to design and acquire the special Japanese typewriters needed to enhance these and the more formal intercept operations yet to come. 

 

            Those accomplishments in turn anticipated the contributions to be made through the technical and analytical skills of carefully selected Navy radiomen.  The special training needed to convert these radiomen into full-fledged intercept operators was largely devised by the radiomen themselves, and in this regard the accomplishment of Chief Radioman Harry Kidder was at least as significant as that of Lieutenant Safford.  Furthermore, without the incentive provided by the enlisted analysts on Guam in 1930, the special effort to test the intelligence potential of traffic analysis might not have come to pass in 1933.  It was in this test that Lieutenant Joseph Wenger proved that traffic analysis is a unique source of special intelligence.  He was the father of traffic analysis in the Navy, in much the same sense that Lieutenant Safford was the father of cryptanalysis.  Remarkably, this protracted development effort had its genesis in the era of "downsizing" that followed World War I, when the allocation of such Navy funds as were available all but ignored the needs of intelligence. 

 

            Then too, the pronouncement in 1929 by the Secretary of State, Henry L. Stimson, that "gentlemen do not read each other's mail" tended to place any monies openly programmed for that purpose in a very unfavorable light.  Had the Navy Department allowed Stimson's ultimatum to blunt the efforts of the visionaries noted above, there would have been no Navy radio intelligence organization, no victory in the Coral Sea or at Midway, and perhaps no decisive victory anywhere else. 

 

            In the present era, one can but hope that strategic and tactical intelligence needs will never again be allowed to hang by such a slender thread.  Budgetary planners, both military and civilian, need to realize that the next shooting war could well be lost if the silent war has not been won in peacetime. 

 

Notes

 

1.  Laurence F. Safford, "A Brief History of Communications Intelligence in the United States," Special Research History, National Archives, Modern Military Branch, Military Archives Division [hereafter SRH no. 149, 21-7 March 1952, p. 6; and Laurance F. Safford, "History of Radio Intelligence: The Undeclared War," SRH-305, 15 November 1943, pp. 03-05.

 

2.  Jack S. Holtwick, Jr.  (Capt., USN), comp., "Naval Security Group History to World War II SRH-355, June 197 1, pp. 39-40. 

 

3.  P. L. Phillips, "Kidder First OTRG Instructor," Cryptolog, Summer 1983, p. 9; and Edwin T. Layton with Roger Pineau and John Costello, And I Was There (New York: Morrow, 1985), p. 74.

 

4.  Holtwick, SRH-355, pp. 40--l.

 

5.  Ibid., pp. 37-40; and "A Brief History of the Radio Security Station Fourth Marine Regiment, Shanghai, China," SRH-179, p. 2. 

 

6.  "A Brief History of the Radio Security Station Marine Detachment, Peiping, China," SRH-l" pp. 6-8; and SRH- 179, pp. 8-11. 

 

7.  Safford, SRH-305, pp. 03-05. 

 

8.  Phillips, p. 9.

 

9.  Safford, SRH-305, p. 06. 

 

10.  Safford, SRH-149, p. 12; and Safford, SRH-305, p. 06. 

 

11.  Safford, SRH-149, p. 12; and Safford, SRH-305, pp. 09-12. 

 

12.  Safford, SRH-305, pp. 14-6. 

 

13.  Safford, SRH 149, p. 12; and Safford, SRH-305, p. 19. 

 

14.  "A Brief History of U.S. Naval Pre-World War Il Radio Intelligence Activities in the Philippine Islands," SRH-180, P. 3. 

 

15.  SRH-179, p. 24. 

 

16.  Safford, SRH-149, p. 15. 

 

17.  Frederick D. Parker, "The Unsolved Messages of Pearl Harbor," Cryptologia, October 1991 pp. 297-9. 

 

18.  Ibid., p. 299. 

 

19.  Informal discussion with Roger Pineau, October 1992. 

 

20.  John Winton, Ultra in the Pacific: How Breaking Japanese Codes and Ciphers Affected Naval Operations Against Japan, 1941-45 (Annapolis, Md.: U.S. Naval Institute Press, 1994). 

 

21.  Layton, p. 474.

22.  Holtwick, SRH-355, p. 39. 

 

Return to Top