Coastal bays are important carbon sink areas. The productivity and community structure of the main primary producer, phytoplankton, are significantly affected by human activities. Understanding the long-term change patterns of phytoplankton productivity and community structure in typical bays is an important basis for clarifying the influence of phytoplankton on the carbon sink function of bays. Sanggou Bay is one of the most representative farming areas in China. However, the long-term change patterns and controlling factors of phytoplankton productivity and community structure under different aquaculture conditions in Sanggou Bay are still unclear. In this study, we analyzed sediment cores at four stations in Sanggou Bay (i.e., stations B2 and C2 in the shellfish culture area, station B4 in the shellfish-kelp integrated culture area, and station A6 in the area without aquaculture) to investigate the long-term change patterns and controlling factors of phytoplankton productivity and community structure. Our results showed that phytoplankton productivity in the shellfish culture area showed an increasing trend, which might be related to the terrestrial input and the increased activities of shellfish culture (accelerated dissolved inorganic nitrogen (DIN) cycling). Phytoplankton productivity in the shellfish-kelp integrated culture area showed a hump-shaped change, which was significantly influenced by the change in DIN (p < 0.05) and the integrated culture of shellfish and kelp. Phytoplankton productivity in the area without aquaculture was relatively low, while it showed an upward trend from the 5 cm sediment layer to the surface, which might be attributed to the increased DIN and changes in hydrodynamics. Overall, there were significant regional differences in phytoplankton productivity, that is, phytoplankton productivity was significantly higher in the shellfish-kelp integrated culture area than the area without aquaculture during the last 40 years (p < 0.05), and it was significantly higher in the shellfish-kelp integrated culture area than the shellfish culture area during the last 10 years (p < 0.05). Our results indicate that the shellfish-kelp integrated culture model has significant potential for oceanic organic carbon sink. In addition, the productivity ratio of diatoms to dinoflagellates in Sanggou Bay showed an overall upward trend, which may be influenced by the decreased ratio of nitrogen to phosphorus (N/P), the increased ratio of silicon to nitrogen (Si/N), the increased dissolved organic phosphorus (DOP), the decreased temperature, aquaculture activities, and hydrodynamic changes. The above findings provide a scientific basis for further elucidating the carbon sink potential of aquaculture bays.