Antibacterial materials are widely used to prevent hospital-acquired infections. In our previous report, metal (calcium, copper or zinc)-doped raw silk fabrics were shown to possess strong antibacterial activities against Escherichia coli. However, antibacterial materials may occasionally be harmful to the human body; thus, in this study, we investigated the cytotoxicities of extracts from metal-doped raw silk fabrics with respect to fibroblasts and osteoblasts indirectly. Calcium-doped raw silk fabric demonstrated cytocompatibility with fibroblasts. Contrarily, copper- and zinc-doped raw silk fabrics remarkably decreased the cell densities of fibroblasts, indicating their cytotoxic effects. This observation could be attributed to the high concentrations of the released copper or zinc ions. However, calcium-, copper- and zinc-doped raw silk fabrics did not demonstrate any cytotoxic effects on osteoblasts because a high concentration of the serum alleviated the effects of these metal ions released from the fabrics. Thus, calcium-doped raw silk fabric is a promising antibacterial material that does not induce strong cytotoxicity. This study will facilitate the design of materials that are both antibacterial and safe.