This is a shot in the dark, but could you be thinking of the Needham-Schroeder protocol? It was published in 1978 [1], and an attack was published as much as 18 years later, in 1996 [2]. It is not an encryption method, though, but a protocol. In fact, the original paper does not even specify an encryption method to be used, but uses encryption symbolically. Accordingly, the discovered flaw has nothing to do with encryption; rather, it is a conceptual flaw in the protocol's design. However, it is often used in cryptography/security courses as a historical example of a protocol broken late after its invention, so I thought you might have been thinking of this one.
Another popular example that comes to mind is WEP, a link-layer security protocol which was part of the original IEEE 802.11 specification from 1997 and the predecessor of WPA/WPA2. This one was broken due to cryptographic breaches resulting from various weaknesses in its design and misapplications of cryptographic primitives [3,4].
- [1] Roger M. Needham, Michael D. Schroeder. Using Encryption for Authentication in Large Networks of Computers. Commun. ACM 21(12): 993-999 (1978)
- [2] Gavin Lowe. Breaking and Fixing the Needham-Schroeder Public-Key Protocol Using FDR. TACAS 1996: 147-166
- [3] Nikita Borisov, Ian Goldberg, David Wagner. Intercepting mobile communications: the insecurity of 802.11. MOBICOM 2001: 180-189
- [4] Scott R. Fluhrer, Itsik Mantin, Adi Shamir. Weaknesses in the Key Scheduling Algorithm of RC4. Selected Areas in Cryptography 2001: 1-24